2023-04-13 14:53:31,765 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82 2023-04-13 14:53:31,779 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-13 14:53:31,794 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-13 14:53:31,794 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/cluster_688f3769-b6e4-94a5-b788-510dc932696f, deleteOnExit=true 2023-04-13 14:53:31,794 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-13 14:53:31,795 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/test.cache.data in system properties and HBase conf 2023-04-13 14:53:31,795 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/hadoop.tmp.dir in system properties and HBase conf 2023-04-13 14:53:31,796 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/hadoop.log.dir in system properties and HBase conf 2023-04-13 14:53:31,796 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-13 14:53:31,797 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-13 14:53:31,797 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-13 14:53:31,919 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-13 14:53:32,345 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-13 14:53:32,350 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-13 14:53:32,350 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-13 14:53:32,351 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-13 14:53:32,351 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-13 14:53:32,351 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-13 14:53:32,352 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-13 14:53:32,352 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-13 14:53:32,352 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-13 14:53:32,353 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-13 14:53:32,353 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/nfs.dump.dir in system properties and HBase conf 2023-04-13 14:53:32,354 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/java.io.tmpdir in system properties and HBase conf 2023-04-13 14:53:32,354 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-13 14:53:32,354 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-13 14:53:32,355 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-13 14:53:32,937 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-13 14:53:32,942 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-13 14:53:33,260 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-13 14:53:33,448 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-13 14:53:33,468 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-13 14:53:33,506 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-13 14:53:33,538 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/java.io.tmpdir/Jetty_localhost_39507_hdfs____.53b5hc/webapp 2023-04-13 14:53:33,696 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39507 2023-04-13 14:53:33,733 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-13 14:53:33,733 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-13 14:53:34,156 WARN [Listener at localhost/38971] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-13 14:53:34,301 WARN [Listener at localhost/38971] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-13 14:53:34,326 WARN [Listener at localhost/38971] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-13 14:53:34,334 INFO [Listener at localhost/38971] log.Slf4jLog(67): jetty-6.1.26 2023-04-13 14:53:34,342 INFO [Listener at localhost/38971] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/java.io.tmpdir/Jetty_localhost_41741_datanode____.14xcn6/webapp 2023-04-13 14:53:34,450 INFO [Listener at localhost/38971] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41741 2023-04-13 14:53:34,901 WARN [Listener at localhost/37101] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-13 14:53:35,478 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4557a1cbe750d9a9: Processing first storage report for DS-c31463cb-20a2-4320-9688-2519f01a13ef from datanode 3aa5f5e1-9f33-4bd6-908d-6f9dfe621103 2023-04-13 14:53:35,481 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4557a1cbe750d9a9: from storage DS-c31463cb-20a2-4320-9688-2519f01a13ef node DatanodeRegistration(127.0.0.1:42153, datanodeUuid=3aa5f5e1-9f33-4bd6-908d-6f9dfe621103, infoPort=46161, infoSecurePort=0, ipcPort=37101, storageInfo=lv=-57;cid=testClusterID;nsid=1318012654;c=1681397613026), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-13 14:53:35,481 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4557a1cbe750d9a9: Processing first storage report for DS-6e0f565b-ec9c-4f98-957b-437d2fdca545 from datanode 3aa5f5e1-9f33-4bd6-908d-6f9dfe621103 2023-04-13 14:53:35,482 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4557a1cbe750d9a9: from storage DS-6e0f565b-ec9c-4f98-957b-437d2fdca545 node DatanodeRegistration(127.0.0.1:42153, datanodeUuid=3aa5f5e1-9f33-4bd6-908d-6f9dfe621103, infoPort=46161, infoSecurePort=0, ipcPort=37101, storageInfo=lv=-57;cid=testClusterID;nsid=1318012654;c=1681397613026), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-13 14:53:35,540 DEBUG [Listener at localhost/37101] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82 2023-04-13 14:53:35,609 INFO [Listener at localhost/37101] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/cluster_688f3769-b6e4-94a5-b788-510dc932696f/zookeeper_0, clientPort=62626, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/cluster_688f3769-b6e4-94a5-b788-510dc932696f/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/cluster_688f3769-b6e4-94a5-b788-510dc932696f/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-13 14:53:35,623 INFO [Listener at localhost/37101] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=62626 2023-04-13 14:53:35,633 INFO [Listener at localhost/37101] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 14:53:35,636 INFO [Listener at localhost/37101] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 14:53:36,287 INFO [Listener at localhost/37101] util.FSUtils(471): Created version file at hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b with version=8 2023-04-13 14:53:36,288 INFO [Listener at localhost/37101] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/hbase-staging 2023-04-13 14:53:36,604 INFO [Listener at localhost/37101] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-13 14:53:37,068 INFO [Listener at localhost/37101] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-13 14:53:37,100 INFO [Listener at localhost/37101] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-13 14:53:37,100 INFO [Listener at localhost/37101] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-13 14:53:37,101 INFO [Listener at localhost/37101] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-13 14:53:37,101 INFO [Listener at localhost/37101] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-13 14:53:37,101 INFO [Listener at localhost/37101] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-13 14:53:37,246 INFO [Listener at localhost/37101] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-13 14:53:37,322 DEBUG [Listener at localhost/37101] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-13 14:53:37,434 INFO [Listener at localhost/37101] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46829 2023-04-13 14:53:37,446 INFO [Listener at localhost/37101] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 14:53:37,449 INFO [Listener at localhost/37101] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 14:53:37,476 INFO [Listener at localhost/37101] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46829 connecting to ZooKeeper ensemble=127.0.0.1:62626 2023-04-13 14:53:37,515 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:468290x0, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-13 14:53:37,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:46829-0x1017e0375af0000 connected 2023-04-13 14:53:37,546 DEBUG [Listener at localhost/37101] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-13 14:53:37,547 DEBUG [Listener at localhost/37101] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-13 14:53:37,550 DEBUG [Listener at localhost/37101] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-13 14:53:37,558 DEBUG [Listener at localhost/37101] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46829 2023-04-13 14:53:37,558 DEBUG [Listener at localhost/37101] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46829 2023-04-13 14:53:37,559 DEBUG [Listener at localhost/37101] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46829 2023-04-13 14:53:37,559 DEBUG [Listener at localhost/37101] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46829 2023-04-13 14:53:37,560 DEBUG [Listener at localhost/37101] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46829 2023-04-13 14:53:37,568 INFO [Listener at localhost/37101] master.HMaster(444): hbase.rootdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b, hbase.cluster.distributed=false 2023-04-13 14:53:37,639 INFO [Listener at localhost/37101] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-13 14:53:37,639 INFO [Listener at localhost/37101] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-13 14:53:37,639 INFO [Listener at localhost/37101] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-13 14:53:37,640 INFO [Listener at localhost/37101] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-13 14:53:37,640 INFO [Listener at localhost/37101] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-13 14:53:37,640 INFO [Listener at localhost/37101] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-13 14:53:37,645 INFO [Listener at localhost/37101] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-13 14:53:37,647 INFO [Listener at localhost/37101] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43875 2023-04-13 14:53:37,650 INFO [Listener at localhost/37101] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-13 14:53:37,656 DEBUG [Listener at localhost/37101] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-13 14:53:37,656 INFO [Listener at localhost/37101] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 14:53:37,659 INFO [Listener at localhost/37101] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 14:53:37,661 INFO [Listener at localhost/37101] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43875 connecting to ZooKeeper ensemble=127.0.0.1:62626 2023-04-13 14:53:37,664 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:438750x0, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-13 14:53:37,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:43875-0x1017e0375af0001 connected 2023-04-13 14:53:37,665 DEBUG [Listener at localhost/37101] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-13 14:53:37,667 DEBUG [Listener at localhost/37101] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-13 14:53:37,667 DEBUG [Listener at localhost/37101] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-13 14:53:37,668 DEBUG [Listener at localhost/37101] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43875 2023-04-13 14:53:37,671 DEBUG [Listener at localhost/37101] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43875 2023-04-13 14:53:37,671 DEBUG [Listener at localhost/37101] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43875 2023-04-13 14:53:37,672 DEBUG [Listener at localhost/37101] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43875 2023-04-13 14:53:37,672 DEBUG [Listener at localhost/37101] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43875 2023-04-13 14:53:37,674 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,46829,1681397616434 2023-04-13 14:53:37,683 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-13 14:53:37,685 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,46829,1681397616434 2023-04-13 14:53:37,704 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-13 14:53:37,704 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-13 14:53:37,704 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 14:53:37,706 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-13 14:53:37,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-13 14:53:37,707 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,46829,1681397616434 from backup master directory 2023-04-13 14:53:37,711 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,46829,1681397616434 2023-04-13 14:53:37,711 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-13 14:53:37,712 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-13 14:53:37,712 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,46829,1681397616434 2023-04-13 14:53:37,716 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-13 14:53:37,718 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-13 14:53:37,816 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/hbase.id with ID: 7c259b6f-0aef-40df-abbf-3c7cf173bce1 2023-04-13 14:53:37,862 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 14:53:37,880 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 14:53:37,925 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0307b598 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:53:37,957 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a6fa4cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:53:37,983 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 14:53:37,985 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-13 14:53:38,005 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-13 14:53:38,005 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-13 14:53:38,008 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:53:38,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:53:38,014 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-13 14:53:38,057 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/MasterData/data/master/store-tmp 2023-04-13 14:53:38,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:53:38,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-13 14:53:38,085 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 14:53:38,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 14:53:38,086 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-13 14:53:38,086 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 14:53:38,086 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 14:53:38,086 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-13 14:53:38,087 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/MasterData/WALs/jenkins-hbase4.apache.org,46829,1681397616434 2023-04-13 14:53:38,111 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46829%2C1681397616434, suffix=, logDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/MasterData/WALs/jenkins-hbase4.apache.org,46829,1681397616434, archiveDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/MasterData/oldWALs, maxLogs=10 2023-04-13 14:53:38,173 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42153,DS-c31463cb-20a2-4320-9688-2519f01a13ef,DISK] 2023-04-13 14:53:38,183 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:53:38,250 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/MasterData/WALs/jenkins-hbase4.apache.org,46829,1681397616434/jenkins-hbase4.apache.org%2C46829%2C1681397616434.1681397618122 2023-04-13 14:53:38,251 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42153,DS-c31463cb-20a2-4320-9688-2519f01a13ef,DISK]] 2023-04-13 14:53:38,252 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-13 14:53:38,252 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:53:38,256 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-13 14:53:38,257 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-13 14:53:38,318 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-13 14:53:38,325 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-13 14:53:38,348 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-13 14:53:38,360 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:53:38,367 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-13 14:53:38,369 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-13 14:53:38,387 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-13 14:53:38,392 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 14:53:38,393 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60397015, jitterRate=-0.1000143438577652}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-13 14:53:38,393 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-13 14:53:38,394 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-13 14:53:38,414 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-13 14:53:38,415 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-13 14:53:38,418 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-13 14:53:38,420 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-13 14:53:38,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 36 msec 2023-04-13 14:53:38,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-13 14:53:38,486 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-13 14:53:38,492 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-13 14:53:38,518 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-13 14:53:38,522 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-13 14:53:38,524 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-13 14:53:38,530 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-13 14:53:38,534 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-13 14:53:38,537 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 14:53:38,538 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-13 14:53:38,539 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-13 14:53:38,554 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-13 14:53:38,560 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-13 14:53:38,560 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-13 14:53:38,560 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 14:53:38,560 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,46829,1681397616434, sessionid=0x1017e0375af0000, setting cluster-up flag (Was=false) 2023-04-13 14:53:38,580 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 14:53:38,585 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-13 14:53:38,587 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46829,1681397616434 2023-04-13 14:53:38,593 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 14:53:38,599 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-13 14:53:38,601 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46829,1681397616434 2023-04-13 14:53:38,603 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.hbase-snapshot/.tmp 2023-04-13 14:53:38,677 INFO [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(952): ClusterId : 7c259b6f-0aef-40df-abbf-3c7cf173bce1 2023-04-13 14:53:38,683 DEBUG [RS:0;jenkins-hbase4:43875] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-13 14:53:38,690 DEBUG [RS:0;jenkins-hbase4:43875] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-13 14:53:38,690 DEBUG [RS:0;jenkins-hbase4:43875] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-13 14:53:38,694 DEBUG [RS:0;jenkins-hbase4:43875] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-13 14:53:38,695 DEBUG [RS:0;jenkins-hbase4:43875] zookeeper.ReadOnlyZKClient(139): Connect 0x3eff0014 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:53:38,704 DEBUG [RS:0;jenkins-hbase4:43875] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3038e9bb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:53:38,705 DEBUG [RS:0;jenkins-hbase4:43875] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34076b64, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-13 14:53:38,731 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-13 14:53:38,737 DEBUG [RS:0;jenkins-hbase4:43875] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:43875 2023-04-13 14:53:38,741 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-13 14:53:38,741 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-13 14:53:38,741 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-13 14:53:38,741 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-13 14:53:38,741 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-13 14:53:38,741 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 14:53:38,741 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-13 14:53:38,741 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 14:53:38,742 INFO [RS:0;jenkins-hbase4:43875] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-13 14:53:38,742 INFO [RS:0;jenkins-hbase4:43875] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-13 14:53:38,742 DEBUG [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1023): About to register with Master. 2023-04-13 14:53:38,743 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681397648743 2023-04-13 14:53:38,746 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-13 14:53:38,746 INFO [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46829,1681397616434 with isa=jenkins-hbase4.apache.org/172.31.14.131:43875, startcode=1681397617638 2023-04-13 14:53:38,753 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-13 14:53:38,754 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-13 14:53:38,760 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-13 14:53:38,760 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-13 14:53:38,768 DEBUG [RS:0;jenkins-hbase4:43875] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-13 14:53:38,769 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-13 14:53:38,770 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-13 14:53:38,770 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-13 14:53:38,770 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-13 14:53:38,771 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-13 14:53:38,772 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-13 14:53:38,774 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-13 14:53:38,774 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-13 14:53:38,777 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-13 14:53:38,777 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-13 14:53:38,782 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681397618779,5,FailOnTimeoutGroup] 2023-04-13 14:53:38,784 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681397618783,5,FailOnTimeoutGroup] 2023-04-13 14:53:38,784 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-13 14:53:38,784 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-13 14:53:38,785 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-13 14:53:38,785 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-13 14:53:38,834 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50449, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-13 14:53:38,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:38,863 DEBUG [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b 2023-04-13 14:53:38,863 DEBUG [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:38971 2023-04-13 14:53:38,864 DEBUG [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-13 14:53:38,874 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-13 14:53:38,875 DEBUG [RS:0;jenkins-hbase4:43875] zookeeper.ZKUtil(162): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:38,875 WARN [RS:0;jenkins-hbase4:43875] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-13 14:53:38,875 INFO [RS:0;jenkins-hbase4:43875] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-13 14:53:38,876 DEBUG [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1947): logDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/WALs/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:38,877 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43875,1681397617638] 2023-04-13 14:53:38,886 DEBUG [RS:0;jenkins-hbase4:43875] zookeeper.ZKUtil(162): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:38,896 DEBUG [RS:0;jenkins-hbase4:43875] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-13 14:53:38,905 INFO [RS:0;jenkins-hbase4:43875] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-13 14:53:38,924 INFO [RS:0;jenkins-hbase4:43875] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-13 14:53:38,929 INFO [RS:0;jenkins-hbase4:43875] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-13 14:53:38,929 INFO [RS:0;jenkins-hbase4:43875] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-13 14:53:38,932 INFO [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-13 14:53:38,939 INFO [RS:0;jenkins-hbase4:43875] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-13 14:53:38,939 DEBUG [RS:0;jenkins-hbase4:43875] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 14:53:38,940 DEBUG [RS:0;jenkins-hbase4:43875] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 14:53:38,940 DEBUG [RS:0;jenkins-hbase4:43875] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 14:53:38,940 DEBUG [RS:0;jenkins-hbase4:43875] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 14:53:38,940 DEBUG [RS:0;jenkins-hbase4:43875] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 14:53:38,940 DEBUG [RS:0;jenkins-hbase4:43875] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-13 14:53:38,940 DEBUG [RS:0;jenkins-hbase4:43875] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 14:53:38,940 DEBUG [RS:0;jenkins-hbase4:43875] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 14:53:38,940 DEBUG [RS:0;jenkins-hbase4:43875] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 14:53:38,941 DEBUG [RS:0;jenkins-hbase4:43875] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 14:53:38,944 INFO [RS:0;jenkins-hbase4:43875] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-13 14:53:38,944 INFO [RS:0;jenkins-hbase4:43875] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-13 14:53:38,944 INFO [RS:0;jenkins-hbase4:43875] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-13 14:53:38,967 INFO [RS:0;jenkins-hbase4:43875] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-13 14:53:38,970 INFO [RS:0;jenkins-hbase4:43875] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43875,1681397617638-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-13 14:53:38,994 INFO [RS:0;jenkins-hbase4:43875] regionserver.Replication(203): jenkins-hbase4.apache.org,43875,1681397617638 started 2023-04-13 14:53:38,994 INFO [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,43875,1681397617638, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43875, sessionid=0x1017e0375af0001 2023-04-13 14:53:38,994 DEBUG [RS:0;jenkins-hbase4:43875] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-13 14:53:38,994 DEBUG [RS:0;jenkins-hbase4:43875] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:38,994 DEBUG [RS:0;jenkins-hbase4:43875] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43875,1681397617638' 2023-04-13 14:53:38,994 DEBUG [RS:0;jenkins-hbase4:43875] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:38,995 DEBUG [RS:0;jenkins-hbase4:43875] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:38,996 DEBUG [RS:0;jenkins-hbase4:43875] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-13 14:53:38,996 DEBUG [RS:0;jenkins-hbase4:43875] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-13 14:53:38,996 DEBUG [RS:0;jenkins-hbase4:43875] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:38,996 DEBUG [RS:0;jenkins-hbase4:43875] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43875,1681397617638' 2023-04-13 14:53:38,996 DEBUG [RS:0;jenkins-hbase4:43875] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-13 14:53:38,997 DEBUG [RS:0;jenkins-hbase4:43875] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-13 14:53:38,997 DEBUG [RS:0;jenkins-hbase4:43875] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-13 14:53:38,998 INFO [RS:0;jenkins-hbase4:43875] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-13 14:53:38,998 INFO [RS:0;jenkins-hbase4:43875] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-13 14:53:39,109 INFO [RS:0;jenkins-hbase4:43875] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43875%2C1681397617638, suffix=, logDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/WALs/jenkins-hbase4.apache.org,43875,1681397617638, archiveDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/oldWALs, maxLogs=32 2023-04-13 14:53:39,128 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42153,DS-c31463cb-20a2-4320-9688-2519f01a13ef,DISK] 2023-04-13 14:53:39,132 INFO [RS:0;jenkins-hbase4:43875] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/WALs/jenkins-hbase4.apache.org,43875,1681397617638/jenkins-hbase4.apache.org%2C43875%2C1681397617638.1681397619111 2023-04-13 14:53:39,132 DEBUG [RS:0;jenkins-hbase4:43875] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42153,DS-c31463cb-20a2-4320-9688-2519f01a13ef,DISK]] 2023-04-13 14:53:39,198 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-13 14:53:39,199 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-13 14:53:39,200 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b 2023-04-13 14:53:39,616 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:53:39,619 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-13 14:53:39,621 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/info 2023-04-13 14:53:39,622 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-13 14:53:39,624 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:53:39,624 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-13 14:53:39,626 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/rep_barrier 2023-04-13 14:53:39,627 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-13 14:53:39,628 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:53:39,628 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-13 14:53:39,630 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/table 2023-04-13 14:53:39,631 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-13 14:53:39,632 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:53:39,634 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740 2023-04-13 14:53:39,635 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740 2023-04-13 14:53:39,639 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 14:53:39,642 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-13 14:53:39,646 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 14:53:39,647 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74054458, jitterRate=0.10349741578102112}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 14:53:39,648 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-13 14:53:39,648 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-13 14:53:39,648 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-13 14:53:39,648 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-13 14:53:39,648 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-13 14:53:39,648 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-13 14:53:39,649 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-13 14:53:39,649 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-13 14:53:39,654 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-13 14:53:39,654 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-13 14:53:39,663 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-13 14:53:39,676 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-13 14:53:39,680 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-13 14:53:39,833 DEBUG [jenkins-hbase4:46829] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-13 14:53:39,836 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43875,1681397617638, state=OPENING 2023-04-13 14:53:39,843 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-13 14:53:39,845 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 14:53:39,845 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-13 14:53:39,848 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:53:40,038 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:40,042 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-13 14:53:40,046 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44072, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-13 14:53:40,057 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-13 14:53:40,057 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-13 14:53:40,060 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43875%2C1681397617638.meta, suffix=.meta, logDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/WALs/jenkins-hbase4.apache.org,43875,1681397617638, archiveDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/oldWALs, maxLogs=32 2023-04-13 14:53:40,082 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42153,DS-c31463cb-20a2-4320-9688-2519f01a13ef,DISK] 2023-04-13 14:53:40,086 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/WALs/jenkins-hbase4.apache.org,43875,1681397617638/jenkins-hbase4.apache.org%2C43875%2C1681397617638.meta.1681397620062.meta 2023-04-13 14:53:40,086 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42153,DS-c31463cb-20a2-4320-9688-2519f01a13ef,DISK]] 2023-04-13 14:53:40,087 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-13 14:53:40,089 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-13 14:53:40,108 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-13 14:53:40,113 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-13 14:53:40,119 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-13 14:53:40,119 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:53:40,119 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-13 14:53:40,119 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-13 14:53:40,122 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-13 14:53:40,124 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/info 2023-04-13 14:53:40,124 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/info 2023-04-13 14:53:40,125 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-13 14:53:40,126 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:53:40,126 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-13 14:53:40,128 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/rep_barrier 2023-04-13 14:53:40,128 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/rep_barrier 2023-04-13 14:53:40,129 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-13 14:53:40,130 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:53:40,130 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-13 14:53:40,132 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/table 2023-04-13 14:53:40,132 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/table 2023-04-13 14:53:40,132 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-13 14:53:40,133 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:53:40,135 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740 2023-04-13 14:53:40,138 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740 2023-04-13 14:53:40,142 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 14:53:40,144 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-13 14:53:40,146 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66321941, jitterRate=-0.01172606647014618}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 14:53:40,146 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-13 14:53:40,158 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681397620028 2023-04-13 14:53:40,180 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-13 14:53:40,182 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43875,1681397617638, state=OPEN 2023-04-13 14:53:40,182 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-13 14:53:40,185 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-13 14:53:40,185 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-13 14:53:40,189 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-13 14:53:40,189 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43875,1681397617638 in 337 msec 2023-04-13 14:53:40,194 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-13 14:53:40,195 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 528 msec 2023-04-13 14:53:40,199 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5440 sec 2023-04-13 14:53:40,200 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681397620200, completionTime=-1 2023-04-13 14:53:40,200 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-13 14:53:40,200 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-13 14:53:40,264 DEBUG [hconnection-0xe3d782f-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:53:40,268 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44086, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:53:40,286 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-13 14:53:40,286 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681397680286 2023-04-13 14:53:40,286 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681397740286 2023-04-13 14:53:40,287 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 86 msec 2023-04-13 14:53:40,318 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46829,1681397616434-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-13 14:53:40,319 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46829,1681397616434-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-13 14:53:40,319 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46829,1681397616434-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-13 14:53:40,320 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:46829, period=300000, unit=MILLISECONDS is enabled. 2023-04-13 14:53:40,321 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-13 14:53:40,327 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-13 14:53:40,335 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-13 14:53:40,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-13 14:53:40,347 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-13 14:53:40,350 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 14:53:40,352 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 14:53:40,376 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/hbase/namespace/f80e0880801c37895d7e91109ae26c79 2023-04-13 14:53:40,379 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/hbase/namespace/f80e0880801c37895d7e91109ae26c79 empty. 2023-04-13 14:53:40,380 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/hbase/namespace/f80e0880801c37895d7e91109ae26c79 2023-04-13 14:53:40,380 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-13 14:53:40,405 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-13 14:53:40,409 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => f80e0880801c37895d7e91109ae26c79, NAME => 'hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp 2023-04-13 14:53:40,426 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:53:40,426 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing f80e0880801c37895d7e91109ae26c79, disabling compactions & flushes 2023-04-13 14:53:40,426 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79. 2023-04-13 14:53:40,426 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79. 2023-04-13 14:53:40,426 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79. after waiting 0 ms 2023-04-13 14:53:40,426 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79. 2023-04-13 14:53:40,426 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79. 2023-04-13 14:53:40,426 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for f80e0880801c37895d7e91109ae26c79: 2023-04-13 14:53:40,431 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 14:53:40,447 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681397620435"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397620435"}]},"ts":"1681397620435"} 2023-04-13 14:53:40,475 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 14:53:40,477 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 14:53:40,481 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397620477"}]},"ts":"1681397620477"} 2023-04-13 14:53:40,486 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-13 14:53:40,497 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=f80e0880801c37895d7e91109ae26c79, ASSIGN}] 2023-04-13 14:53:40,500 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=f80e0880801c37895d7e91109ae26c79, ASSIGN 2023-04-13 14:53:40,501 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=f80e0880801c37895d7e91109ae26c79, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43875,1681397617638; forceNewPlan=false, retain=false 2023-04-13 14:53:40,652 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=f80e0880801c37895d7e91109ae26c79, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:40,653 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681397620652"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397620652"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397620652"}]},"ts":"1681397620652"} 2023-04-13 14:53:40,657 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure f80e0880801c37895d7e91109ae26c79, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:53:40,818 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79. 2023-04-13 14:53:40,819 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f80e0880801c37895d7e91109ae26c79, NAME => 'hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79.', STARTKEY => '', ENDKEY => ''} 2023-04-13 14:53:40,820 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace f80e0880801c37895d7e91109ae26c79 2023-04-13 14:53:40,820 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:53:40,820 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f80e0880801c37895d7e91109ae26c79 2023-04-13 14:53:40,820 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f80e0880801c37895d7e91109ae26c79 2023-04-13 14:53:40,822 INFO [StoreOpener-f80e0880801c37895d7e91109ae26c79-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region f80e0880801c37895d7e91109ae26c79 2023-04-13 14:53:40,825 DEBUG [StoreOpener-f80e0880801c37895d7e91109ae26c79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/namespace/f80e0880801c37895d7e91109ae26c79/info 2023-04-13 14:53:40,825 DEBUG [StoreOpener-f80e0880801c37895d7e91109ae26c79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/namespace/f80e0880801c37895d7e91109ae26c79/info 2023-04-13 14:53:40,825 INFO [StoreOpener-f80e0880801c37895d7e91109ae26c79-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f80e0880801c37895d7e91109ae26c79 columnFamilyName info 2023-04-13 14:53:40,826 INFO [StoreOpener-f80e0880801c37895d7e91109ae26c79-1] regionserver.HStore(310): Store=f80e0880801c37895d7e91109ae26c79/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:53:40,828 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/namespace/f80e0880801c37895d7e91109ae26c79 2023-04-13 14:53:40,829 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/namespace/f80e0880801c37895d7e91109ae26c79 2023-04-13 14:53:40,833 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f80e0880801c37895d7e91109ae26c79 2023-04-13 14:53:40,837 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/namespace/f80e0880801c37895d7e91109ae26c79/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 14:53:40,838 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f80e0880801c37895d7e91109ae26c79; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70004270, jitterRate=0.04314491152763367}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-13 14:53:40,838 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f80e0880801c37895d7e91109ae26c79: 2023-04-13 14:53:40,840 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79., pid=6, masterSystemTime=1681397620811 2023-04-13 14:53:40,844 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79. 2023-04-13 14:53:40,844 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79. 2023-04-13 14:53:40,845 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=f80e0880801c37895d7e91109ae26c79, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:40,846 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681397620844"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681397620844"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681397620844"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681397620844"}]},"ts":"1681397620844"} 2023-04-13 14:53:40,853 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-13 14:53:40,853 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure f80e0880801c37895d7e91109ae26c79, server=jenkins-hbase4.apache.org,43875,1681397617638 in 192 msec 2023-04-13 14:53:40,856 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-13 14:53:40,856 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=f80e0880801c37895d7e91109ae26c79, ASSIGN in 356 msec 2023-04-13 14:53:40,857 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 14:53:40,858 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397620858"}]},"ts":"1681397620858"} 2023-04-13 14:53:40,861 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-13 14:53:40,865 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 14:53:40,868 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 527 msec 2023-04-13 14:53:40,950 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-13 14:53:40,951 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-13 14:53:40,951 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 14:53:40,993 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-13 14:53:41,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-13 14:53:41,017 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 35 msec 2023-04-13 14:53:41,027 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-13 14:53:41,041 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-13 14:53:41,045 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 18 msec 2023-04-13 14:53:41,052 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-13 14:53:41,058 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-13 14:53:41,059 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.347sec 2023-04-13 14:53:41,061 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-13 14:53:41,063 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-13 14:53:41,063 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-13 14:53:41,064 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46829,1681397616434-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-13 14:53:41,065 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46829,1681397616434-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-13 14:53:41,076 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-13 14:53:41,082 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x69cada93 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:53:41,087 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@53e64c30, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:53:41,101 DEBUG [hconnection-0x6468bb18-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:53:41,114 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44096, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:53:41,124 INFO [Listener at localhost/37101] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,46829,1681397616434 2023-04-13 14:53:41,151 INFO [Listener at localhost/37101] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=208, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=275, ProcessCount=169, AvailableMemoryMB=1710 2023-04-13 14:53:41,168 DEBUG [Listener at localhost/37101] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 14:53:41,172 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46300, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 14:53:41,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 14:53:41,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 14:53:41,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 14:53:41,191 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 14:53:41,194 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 14:53:41,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-13 14:53:41,197 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:41,199 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05 empty. 2023-04-13 14:53:41,201 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:41,201 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 14:53:41,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-13 14:53:41,219 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 14:53:41,222 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0d965d81bccf303fd24a24be9d6cbc05, NAME => 'TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp 2023-04-13 14:53:41,239 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:53:41,239 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 0d965d81bccf303fd24a24be9d6cbc05, disabling compactions & flushes 2023-04-13 14:53:41,239 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:41,239 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:41,239 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. after waiting 0 ms 2023-04-13 14:53:41,240 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:41,240 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:41,240 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:41,244 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 14:53:41,246 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397621246"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397621246"}]},"ts":"1681397621246"} 2023-04-13 14:53:41,249 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 14:53:41,251 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 14:53:41,251 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397621251"}]},"ts":"1681397621251"} 2023-04-13 14:53:41,254 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 14:53:41,258 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0d965d81bccf303fd24a24be9d6cbc05, ASSIGN}] 2023-04-13 14:53:41,261 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0d965d81bccf303fd24a24be9d6cbc05, ASSIGN 2023-04-13 14:53:41,263 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0d965d81bccf303fd24a24be9d6cbc05, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43875,1681397617638; forceNewPlan=false, retain=false 2023-04-13 14:53:41,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-13 14:53:41,414 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=0d965d81bccf303fd24a24be9d6cbc05, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:41,415 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397621414"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397621414"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397621414"}]},"ts":"1681397621414"} 2023-04-13 14:53:41,418 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:53:41,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-13 14:53:41,578 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:41,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0d965d81bccf303fd24a24be9d6cbc05, NAME => 'TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.', STARTKEY => '', ENDKEY => ''} 2023-04-13 14:53:41,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:41,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:53:41,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:41,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:41,581 INFO [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:41,583 DEBUG [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A 2023-04-13 14:53:41,584 DEBUG [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A 2023-04-13 14:53:41,585 INFO [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:53:41,586 INFO [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0d965d81bccf303fd24a24be9d6cbc05 columnFamilyName A 2023-04-13 14:53:41,587 INFO [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] regionserver.HStore(310): Store=0d965d81bccf303fd24a24be9d6cbc05/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:53:41,587 INFO [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:41,589 DEBUG [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B 2023-04-13 14:53:41,589 DEBUG [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B 2023-04-13 14:53:41,589 INFO [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:53:41,590 INFO [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0d965d81bccf303fd24a24be9d6cbc05 columnFamilyName B 2023-04-13 14:53:41,590 INFO [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] regionserver.HStore(310): Store=0d965d81bccf303fd24a24be9d6cbc05/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:53:41,591 INFO [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:41,593 DEBUG [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C 2023-04-13 14:53:41,593 DEBUG [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C 2023-04-13 14:53:41,593 INFO [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:53:41,594 INFO [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0d965d81bccf303fd24a24be9d6cbc05 columnFamilyName C 2023-04-13 14:53:41,595 INFO [StoreOpener-0d965d81bccf303fd24a24be9d6cbc05-1] regionserver.HStore(310): Store=0d965d81bccf303fd24a24be9d6cbc05/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:53:41,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:41,597 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:41,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:41,602 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 14:53:41,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:41,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 14:53:41,609 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0d965d81bccf303fd24a24be9d6cbc05; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70008167, jitterRate=0.04320298135280609}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 14:53:41,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:41,611 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., pid=11, masterSystemTime=1681397621572 2023-04-13 14:53:41,614 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:41,614 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:41,615 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=0d965d81bccf303fd24a24be9d6cbc05, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:41,615 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397621615"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681397621615"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681397621615"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681397621615"}]},"ts":"1681397621615"} 2023-04-13 14:53:41,622 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-13 14:53:41,622 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 in 200 msec 2023-04-13 14:53:41,626 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-13 14:53:41,627 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0d965d81bccf303fd24a24be9d6cbc05, ASSIGN in 364 msec 2023-04-13 14:53:41,628 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 14:53:41,628 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397621628"}]},"ts":"1681397621628"} 2023-04-13 14:53:41,631 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 14:53:41,634 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 14:53:41,637 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 447 msec 2023-04-13 14:53:41,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-13 14:53:41,821 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-13 14:53:41,827 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x7d9fc901 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:53:41,832 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44ef43e3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:53:41,838 DEBUG [Listener at localhost/37101] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:53:41,841 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44098, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:53:41,845 DEBUG [Listener at localhost/37101] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 14:53:41,848 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46306, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 14:53:41,858 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x79eb61dd to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:53:41,869 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61e72c30, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:53:41,870 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x748f869b to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:53:41,876 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e16d41b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:53:41,876 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x5e91d243 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:53:41,884 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fea27db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:53:41,884 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x30eb7fb2 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:53:41,889 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e728705, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:53:41,889 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x46192d17 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:53:41,902 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63859c62, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:53:41,904 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x60b8c4c7 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:53:41,918 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49a45b6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:53:41,918 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x49d71120 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:53:41,950 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fef7a21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:53:41,951 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x2879e2e1 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:53:41,956 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cd3abbd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:53:41,957 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x0d24bbb7 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:53:41,961 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6aad2e18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:53:41,975 DEBUG [hconnection-0x6260af1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:53:41,976 DEBUG [hconnection-0x55d4bc5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:53:41,976 DEBUG [hconnection-0x62a93ad6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:53:41,976 DEBUG [hconnection-0x6ff4c9b4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:53:41,980 DEBUG [hconnection-0x7576e4ed-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:53:41,981 DEBUG [hconnection-0xfa92e03-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:53:41,986 DEBUG [hconnection-0x74b74e95-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:53:41,986 DEBUG [hconnection-0x1bd47f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:53:41,987 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44114, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:53:41,988 DEBUG [hconnection-0x26f80c4e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:53:41,990 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44112, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:53:41,993 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44160, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:53:41,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:41,998 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44146, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:53:41,998 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44174, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:53:41,999 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44130, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:53:41,999 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:53:41,999 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44164, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:53:42,000 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44182, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:53:42,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:42,023 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:42,023 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:42,024 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:42,025 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:42,026 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:42,026 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:42,028 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:42,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:42,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:42,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:42,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:42,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:42,059 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:42,061 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,061 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:42,064 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:42,064 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:42,071 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:42,082 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:42,082 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:42,084 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:42,084 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:42,090 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 14:53:42,104 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:42,105 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:42,106 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:42,106 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:42,106 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:42,106 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:42,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:42,117 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:42,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397622162, max timestamp=1681397622202Num uniques 859; 2023-04-13 14:53:42,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:42,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397622162, max timestamp=1681397622202Num uniques 859; 2023-04-13 14:53:42,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-13 14:53:42,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397622162, max timestamp=1681397622205Num uniques 859; 2023-04-13 14:53:42,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-13 14:53:42,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44168 deadline: 1681397682287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9975, totalHeapSize=22912, min timestamp=1681397622274, max timestamp=1681397622302Num uniques 1018; 2023-04-13 14:53:42,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46615, new segment data size=9975, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-13 14:53:42,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397622274, max timestamp=1681397622302Num uniques 1009; 2023-04-13 14:53:42,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-13 14:53:42,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397622274, max timestamp=1681397622302Num uniques 1009; 2023-04-13 14:53:42,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46203, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-13 14:53:42,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397682405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397622389, max timestamp=1681397622407Num uniques 1077; 2023-04-13 14:53:42,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397622389, max timestamp=1681397622407Num uniques 1009; 2023-04-13 14:53:42,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397622389, max timestamp=1681397622404Num uniques 1009; 2023-04-13 14:53:42,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=6870, suffix heap size=139976,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=150 2023-04-13 14:53:42,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46224, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-13 14:53:42,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-13 14:53:42,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681397622445, max timestamp=1681397622458Num uniques 1009; 2023-04-13 14:53:42,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10806, suffix heap size=132904,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=236 2023-04-13 14:53:42,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681397622445, max timestamp=1681397622458Num uniques 1009; 2023-04-13 14:53:42,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10806, suffix heap size=132904,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=236 2023-04-13 14:53:42,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681397622445, max timestamp=1681397622458Num uniques 1009; 2023-04-13 14:53:42,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10806, suffix heap size=132904,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=236 2023-04-13 14:53:42,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44146 deadline: 1681397682495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44168 deadline: 1681397682497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44160 deadline: 1681397682497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397682499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397682518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44146 deadline: 1681397682605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44168 deadline: 1681397682605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397682606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44160 deadline: 1681397682606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,668 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/abd179e50af141b393d084431a796a20 2023-04-13 14:53:42,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397682723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397682815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44168 deadline: 1681397682815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44160 deadline: 1681397682815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:42,824 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/fc979feb74324bf7be8a081277fba9ff 2023-04-13 14:53:42,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:42,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44146 deadline: 1681397682822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:43,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397683035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:43,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397683118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:43,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44168 deadline: 1681397683121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:43,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44160 deadline: 1681397683125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:43,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44146 deadline: 1681397683133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,300 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/1f44c6dacf3e428190873301edc09497 2023-04-13 14:53:43,313 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/abd179e50af141b393d084431a796a20 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/abd179e50af141b393d084431a796a20 2023-04-13 14:53:43,327 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/abd179e50af141b393d084431a796a20, entries=150, sequenceid=10, filesize=11.6 K 2023-04-13 14:53:43,335 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/fc979feb74324bf7be8a081277fba9ff as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/fc979feb74324bf7be8a081277fba9ff 2023-04-13 14:53:43,378 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/fc979feb74324bf7be8a081277fba9ff, entries=150, sequenceid=10, filesize=11.6 K 2023-04-13 14:53:43,381 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/1f44c6dacf3e428190873301edc09497 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/1f44c6dacf3e428190873301edc09497 2023-04-13 14:53:43,391 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/1f44c6dacf3e428190873301edc09497, entries=150, sequenceid=10, filesize=11.6 K 2023-04-13 14:53:43,399 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 0d965d81bccf303fd24a24be9d6cbc05 in 1308ms, sequenceid=10, compaction requested=false 2023-04-13 14:53:43,401 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-13 14:53:43,404 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:43,404 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:43,404 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:43,414 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:43,414 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,414 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:43,414 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:43,417 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,418 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:43,418 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,419 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:43,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:43,421 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:43,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a5fe95c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:43,422 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,427 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,427 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,427 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:43,427 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:43,427 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:43,428 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:43,450 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,450 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:43,451 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:43,451 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:43,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:43,457 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:43,457 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:43,457 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:43,457 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:43,457 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:43,471 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,471 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:43,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:43,471 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:43,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:43,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:43,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:43,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,474 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:43,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,477 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,495 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:43,495 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:43,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:43,496 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:43,496 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:43,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:43,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:43,496 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:43,498 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:43,498 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,498 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,498 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,498 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:43,498 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,499 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:43,551 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=147.60 KB heapSize=418.31 KB 2023-04-13 14:53:43,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:43,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:53:43,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:43,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:53:43,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:43,553 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:53:43,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:43,600 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:43,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:43,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:43,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:43,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:43,632 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:43,632 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:43,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:43,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:43,637 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,637 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:43,639 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,639 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:43,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:43,640 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,640 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:43,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:43,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:43,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:43,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:43,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:43,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:43,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:43,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:43,646 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:43,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:43,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:43,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:43,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:43,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:43,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:43,653 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:43,653 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:43,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:43,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:43,657 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,657 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:43,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:43,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:43,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:43,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:43,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:43,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54283385[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:43,662 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:43,662 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:43,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44168 deadline: 1681397683627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,665 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,665 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:43,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:43,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44160 deadline: 1681397683638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:43,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44146 deadline: 1681397683638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,675 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,675 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,677 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:43,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,677 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:43,677 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:43,677 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:43,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:43,681 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:43,681 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:43,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397683661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:43,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:43,681 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:43,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:43,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397683664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:43,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:43,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:43,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:43,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:43,689 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:43,689 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:43,689 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:43,689 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:43,689 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:43,694 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,694 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,695 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:43,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:43,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:43,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:43,697 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:43,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,701 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,709 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:43,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:43,709 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,709 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:43,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:43,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:43,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:43,709 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:43,709 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:43,713 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:43,713 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,713 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:43,713 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:43,713 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,713 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,713 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:43,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:43,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397683786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:43,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397683787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,813 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:43,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:43,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:43,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:43,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:43,828 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:43,828 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:43,830 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:43,830 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:43,830 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,830 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:43,832 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:43,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:43,834 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,834 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:43,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:43,834 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:43,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:43,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:43,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:43,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:43,836 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:43,836 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:43,836 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:43,836 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:43,837 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:43,837 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:43,837 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:43,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:43,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:43,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:43,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:43,842 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,842 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:43,843 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,843 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:43,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:43,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477fc28f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:43,846 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:43,846 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,847 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,848 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:43,848 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,848 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:43,848 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:43,848 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:43,852 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:43,852 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:43,852 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:43,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:43,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:43,858 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:43,858 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:43,858 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:43,858 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:43,858 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:43,861 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,861 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:43,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,861 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:43,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:43,863 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:43,866 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:43,880 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:43,880 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:43,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:43,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:43,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:43,880 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,882 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:43,882 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:43,882 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:43,880 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:43,882 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:43,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:43,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:43,883 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:43,883 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,884 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:43,884 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:43,983 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:43,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:43,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:44,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:44,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:44,004 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:44,004 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:44,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:44,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:44,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:44,007 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,007 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397683997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:44,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397683998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,009 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:44,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,010 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,011 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,011 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:44,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:44,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:44,028 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/62d5c7f0c3d64d2db3a52747c14757c9 2023-04-13 14:53:44,028 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:44,028 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:44,033 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:44,033 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:44,033 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:44,033 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:44,033 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:44,033 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:44,033 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:44,035 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:44,035 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:44,035 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,035 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:44,035 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:44,038 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,038 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:44,039 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,039 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:44,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:44,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77c09868[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:44,042 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:44,042 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,043 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,044 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:44,044 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:44,044 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:44,045 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,045 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:44,053 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:44,053 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:44,053 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:44,053 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:44,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:44,061 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:44,061 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:44,061 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:44,061 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:44,061 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:44,063 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,063 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,064 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:44,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:44,065 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:44,068 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:44,082 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,082 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,082 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:44,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:44,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:44,082 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,088 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,088 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,088 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,088 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,082 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:44,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:44,088 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:44,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:44,088 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:44,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:44,089 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:44,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:44,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:44,096 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/07bace4873fc4c7089463d70a8d91af1 2023-04-13 14:53:44,156 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/e3fa0d10b5564bcebe6a5efe813a2d61 2023-04-13 14:53:44,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/62d5c7f0c3d64d2db3a52747c14757c9 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/62d5c7f0c3d64d2db3a52747c14757c9 2023-04-13 14:53:44,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/62d5c7f0c3d64d2db3a52747c14757c9, entries=200, sequenceid=99, filesize=13.9 K 2023-04-13 14:53:44,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/07bace4873fc4c7089463d70a8d91af1 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/07bace4873fc4c7089463d70a8d91af1 2023-04-13 14:53:44,189 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:44,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:44,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:44,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:44,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:44,207 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:44,207 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:44,208 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:44,208 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:44,208 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,209 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,212 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,212 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:44,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:44,212 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,213 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:44,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:44,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:44,220 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:44,220 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:44,233 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:44,245 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/07bace4873fc4c7089463d70a8d91af1, entries=150, sequenceid=99, filesize=11.6 K 2023-04-13 14:53:44,247 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/e3fa0d10b5564bcebe6a5efe813a2d61 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/e3fa0d10b5564bcebe6a5efe813a2d61 2023-04-13 14:53:44,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:44,249 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:44,249 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:44,249 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:44,249 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:44,249 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:44,250 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:44,250 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:44,250 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,253 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:44,253 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:44,255 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:44,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:44,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:44,257 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,258 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:44,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:44,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:44,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:44,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62409ac5[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:44,261 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:44,261 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,262 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/e3fa0d10b5564bcebe6a5efe813a2d61, entries=150, sequenceid=99, filesize=11.6 K 2023-04-13 14:53:44,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,268 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,268 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:44,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,269 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~418.27 KB/428304, currentSize=33.54 KB/34350 for 0d965d81bccf303fd24a24be9d6cbc05 in 717ms, sequenceid=99, compaction requested=false 2023-04-13 14:53:44,269 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:44,270 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:44,270 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:44,270 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:44,272 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:44,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:44,279 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:44,279 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:44,279 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:44,279 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:44,279 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:44,281 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,281 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,282 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:44,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:44,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,287 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,296 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,296 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,296 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:44,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:44,296 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:44,296 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,297 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,297 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,297 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:44,298 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:44,298 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:44,298 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:44,298 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:44,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:44,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:44,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:44,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:44,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:44,321 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:53:44,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:44,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:44,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:44,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:44,322 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:44,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:44,397 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/00c5158cf7934c21a1ebfa0cab5efa20 2023-04-13 14:53:44,398 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:44,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:44,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:44,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:44,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:44,430 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:44,430 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:44,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397624386, max timestamp=1681397624419Num uniques 859; 2023-04-13 14:53:44,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:44,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397624386, max timestamp=1681397624424Num uniques 859; 2023-04-13 14:53:44,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397624386, max timestamp=1681397624424Num uniques 859; 2023-04-13 14:53:44,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:44,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:44,437 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:44,437 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:44,438 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,438 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,443 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:44,443 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:44,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:44,446 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,446 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:44,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:44,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:44,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:44,456 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:44,456 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:44,465 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:44,466 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:44,466 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:44,467 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:44,470 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:44,470 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:44,470 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:44,471 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:44,471 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:44,471 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,471 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:44,471 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:44,473 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:44,475 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:44,475 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,475 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:44,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:44,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77613913[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:44,478 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:44,478 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,479 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,479 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:44,479 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:44,479 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:44,481 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,481 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:44,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:44,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:44,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:44,482 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:44,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:44,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:44,487 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:44,488 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:44,488 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:44,488 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:44,488 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:44,495 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,495 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:44,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,495 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:44,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:44,496 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,498 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,503 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:44,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:44,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:44,503 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:44,503 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,503 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,503 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,503 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,503 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,503 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:44,503 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,504 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:44,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:44,505 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:44,505 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:44,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:44,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:44,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:44,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397684514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:44,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397684515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,605 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:44,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:44,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:44,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:44,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:44,614 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:44,614 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:44,616 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:44,616 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:44,617 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,617 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,619 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:44,619 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:44,619 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,619 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:44,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:44,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:44,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397684626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:44,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397684626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:44,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:44,632 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:44,632 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:44,637 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:44,637 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:44,638 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:44,638 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:44,638 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:44,638 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:44,638 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:44,642 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:44,642 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:44,642 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,642 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:44,642 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:44,645 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,645 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:44,645 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,645 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:44,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:44,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:44,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eb3f6ca[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:44,649 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:44,649 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,650 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,650 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:44,650 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:44,650 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:44,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:44,654 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:44,655 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:44,655 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:44,655 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:44,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:44,663 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:44,663 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:44,663 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:44,663 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:44,663 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:44,666 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,667 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:44,667 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,669 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:44,672 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:44,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44168 deadline: 1681397684674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,677 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:44,677 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:44,677 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:44,677 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:44,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:44,677 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:44,677 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,681 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,681 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,681 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,681 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:44,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44146 deadline: 1681397684677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:44,707 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:44,707 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:44,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:44,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44160 deadline: 1681397684707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,807 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:44,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:44,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:44,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:44,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:44,817 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:44,817 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:44,819 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:44,819 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:44,820 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,820 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,821 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,821 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:44,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:44,822 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,822 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:44,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:44,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:44,830 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:44,831 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:44,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:44,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:44,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:44,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:44,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:44,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:44,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:44,836 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:44,837 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:44,837 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,837 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:44,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:44,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d4324ac[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:44,839 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:44,839 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:44,840 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397684835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:44,841 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:44,841 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:44,841 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,841 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:44,842 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:44,843 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,843 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:44,843 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:44,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397684832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:44,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:44,850 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:44,850 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:44,850 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:44,850 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:44,850 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:44,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:44,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:44,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:44,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,855 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:44,859 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,869 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:44,869 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:44,869 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,869 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:44,869 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:44,869 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:44,869 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:44,869 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:44,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:44,871 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/d8f753fabc364190bc2273e9dc5da922 2023-04-13 14:53:44,869 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:44,881 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:44,881 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:44,881 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:44,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:44,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:44,902 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-13 14:53:44,904 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-13 14:53:44,981 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:44,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:44,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:44,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:44,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:44,996 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:44,996 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:44,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:44,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:45,000 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,000 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,005 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:45,006 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:45,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:45,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:45,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:45,014 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:45,014 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:45,017 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:45,017 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:45,017 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:45,018 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:45,018 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:45,018 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:45,018 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:45,020 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,021 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,021 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:45,023 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:45,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:45,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bb31773[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:45,029 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:45,029 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,030 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:45,035 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,035 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:45,035 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:45,036 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:45,036 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:45,037 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:45,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:45,042 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:45,042 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:45,042 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:45,042 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:45,042 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:45,046 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,046 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,046 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:45,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:45,047 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:45,050 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:45,054 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:45,054 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:45,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:45,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:45,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,059 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,059 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,059 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,059 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:45,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:45,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:45,059 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,065 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,065 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,065 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,066 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:45,066 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:45,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:45,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:45,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397685148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:45,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397685150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,166 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:45,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:45,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:45,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:45,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:45,182 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:45,182 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:45,185 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:45,185 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:45,187 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,187 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,188 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,189 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,189 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:45,189 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:45,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:45,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:45,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:45,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:45,196 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:45,196 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:45,198 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:45,200 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:45,200 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:45,200 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:45,200 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:45,200 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:45,200 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:45,200 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:45,200 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:45,200 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,201 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:45,201 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:45,202 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,202 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:45,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:45,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:45,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:45,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:45,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:45,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:45,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25e404b7[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:45,206 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:45,206 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,208 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,208 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:45,208 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:45,209 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:45,209 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,209 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:45,210 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,210 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:45,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:45,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:45,210 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:45,211 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:45,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:45,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:45,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:45,218 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:45,218 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:45,218 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:45,218 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:45,218 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:45,220 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,221 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:45,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,221 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:45,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:45,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:45,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:45,222 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,224 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,227 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:45,227 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:45,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:45,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:45,227 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,227 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,227 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,227 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,227 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,227 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,227 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:45,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:45,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:45,228 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:45,228 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:45,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:45,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:45,331 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:45,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:45,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:45,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:45,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:45,344 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:45,344 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:45,346 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:45,346 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:45,347 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,347 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,347 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/02e5f6ec55be4fbbb7bd29d7dc65f19f 2023-04-13 14:53:45,357 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,357 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,357 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:45,368 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:45,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:45,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:45,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:45,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:45,373 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:45,373 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:45,375 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:45,379 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:45,379 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:45,380 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:45,380 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:45,380 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:45,380 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:45,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:45,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:45,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:45,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:45,382 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:45,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:45,385 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/00c5158cf7934c21a1ebfa0cab5efa20 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/00c5158cf7934c21a1ebfa0cab5efa20 2023-04-13 14:53:45,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:45,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7430a367[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:45,385 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,385 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:45,385 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:45,385 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,387 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:45,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:45,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:45,388 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,388 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:45,390 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,390 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:45,390 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:45,390 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:45,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:45,399 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:45,399 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:45,399 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:45,400 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:45,401 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:45,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,404 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/00c5158cf7934c21a1ebfa0cab5efa20, entries=150, sequenceid=109, filesize=11.6 K 2023-04-13 14:53:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:45,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:45,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,406 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:45,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,412 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/d8f753fabc364190bc2273e9dc5da922 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/d8f753fabc364190bc2273e9dc5da922 2023-04-13 14:53:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:45,412 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:45,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:45,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:45,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:45,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:45,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:45,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:45,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:45,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,418 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:45,418 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,417 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:45,418 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:45,418 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:45,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:45,434 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/d8f753fabc364190bc2273e9dc5da922, entries=150, sequenceid=109, filesize=11.6 K 2023-04-13 14:53:45,441 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/02e5f6ec55be4fbbb7bd29d7dc65f19f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/02e5f6ec55be4fbbb7bd29d7dc65f19f 2023-04-13 14:53:45,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/02e5f6ec55be4fbbb7bd29d7dc65f19f, entries=150, sequenceid=109, filesize=11.6 K 2023-04-13 14:53:45,461 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 0d965d81bccf303fd24a24be9d6cbc05 in 1140ms, sequenceid=109, compaction requested=true 2023-04-13 14:53:45,461 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:45,469 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:45,469 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:45,469 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:45,469 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:53:45,471 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:53:45,476 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:53:45,478 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/B is initiating minor compaction (all files) 2023-04-13 14:53:45,478 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/B in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:45,479 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/fc979feb74324bf7be8a081277fba9ff, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/07bace4873fc4c7089463d70a8d91af1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/d8f753fabc364190bc2273e9dc5da922] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=34.8 K 2023-04-13 14:53:45,482 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting fc979feb74324bf7be8a081277fba9ff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681397622018 2023-04-13 14:53:45,483 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 07bace4873fc4c7089463d70a8d91af1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1681397622480 2023-04-13 14:53:45,483 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:53:45,485 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/A is initiating minor compaction (all files) 2023-04-13 14:53:45,486 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/A in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:45,486 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/abd179e50af141b393d084431a796a20, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/62d5c7f0c3d64d2db3a52747c14757c9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/00c5158cf7934c21a1ebfa0cab5efa20] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=37.1 K 2023-04-13 14:53:45,485 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting d8f753fabc364190bc2273e9dc5da922, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681397623585 2023-04-13 14:53:45,497 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting abd179e50af141b393d084431a796a20, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681397622018 2023-04-13 14:53:45,500 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 62d5c7f0c3d64d2db3a52747c14757c9, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1681397622480 2023-04-13 14:53:45,501 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 00c5158cf7934c21a1ebfa0cab5efa20, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681397623585 2023-04-13 14:53:45,523 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:45,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:45,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:45,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:45,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:45,531 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:45,531 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:45,534 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:45,534 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:45,534 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,534 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,536 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:45,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:45,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:45,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:45,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:45,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:45,547 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:45,565 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:45,565 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:45,565 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-13 14:53:45,566 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:45,566 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:53:45,566 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:45,566 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:53:45,567 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:45,567 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:53:45,587 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#A#compaction#10 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:45,592 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#B#compaction#9 average throughput is 0.21 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:45,696 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:45,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:45,696 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 14:53:45,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:45,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397685732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:45,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397685735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,747 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/e4ab0937a1ea46e7a68475ac0f0e9193 2023-04-13 14:53:45,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:45,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397685841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:45,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397685861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:45,883 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/eb308aa91da84aa79329c8c9627cbbdd 2023-04-13 14:53:45,964 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/bb9ba5c0e5d3498393d0095f3d2cf95a 2023-04-13 14:53:45,978 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/e4ab0937a1ea46e7a68475ac0f0e9193 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/e4ab0937a1ea46e7a68475ac0f0e9193 2023-04-13 14:53:45,990 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/e4ab0937a1ea46e7a68475ac0f0e9193, entries=150, sequenceid=146, filesize=11.8 K 2023-04-13 14:53:46,013 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/eb308aa91da84aa79329c8c9627cbbdd as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/eb308aa91da84aa79329c8c9627cbbdd 2023-04-13 14:53:46,032 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/eb308aa91da84aa79329c8c9627cbbdd, entries=150, sequenceid=146, filesize=11.8 K 2023-04-13 14:53:46,036 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/bb9ba5c0e5d3498393d0095f3d2cf95a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/bb9ba5c0e5d3498393d0095f3d2cf95a 2023-04-13 14:53:46,053 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/bb9ba5c0e5d3498393d0095f3d2cf95a, entries=150, sequenceid=146, filesize=11.8 K 2023-04-13 14:53:46,064 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 0d965d81bccf303fd24a24be9d6cbc05 in 499ms, sequenceid=146, compaction requested=true 2023-04-13 14:53:46,065 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:46,065 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:46,065 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:46,069 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:46,069 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,069 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:46,069 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:46,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:46,073 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 14:53:46,073 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:46,073 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:46,073 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:46,073 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:46,073 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:46,074 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:46,078 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,078 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:46,079 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:46,079 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:46,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:46,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:46,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50f29f09[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:46,081 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:46,081 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,083 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,083 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:46,083 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:46,083 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:46,084 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,084 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:46,085 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:46,085 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:46,085 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:46,085 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:46,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:46,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:46,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:46,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:46,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:46,091 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:46,091 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:46,091 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:46,091 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:46,091 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:46,112 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,112 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:46,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,113 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:46,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:46,114 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:46,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,119 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:46,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:46,119 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:46,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:46,119 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:46,119 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:46,119 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:46,119 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,119 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,120 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,120 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:46,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:46,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:46,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:46,120 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:46,120 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:46,180 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/6defd5a7f3e24303baba8b05a414cc9c 2023-04-13 14:53:46,192 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/4ec279e6356f4eaa8756458ca0cda594 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/4ec279e6356f4eaa8756458ca0cda594 2023-04-13 14:53:46,204 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/50d0691c997a41f7a25f7b7df81245af as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/50d0691c997a41f7a25f7b7df81245af 2023-04-13 14:53:46,221 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:46,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:46,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:46,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:46,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:46,237 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:46,237 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:46,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:46,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:46,242 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,242 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:46,243 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:46,244 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/A of 0d965d81bccf303fd24a24be9d6cbc05 into 4ec279e6356f4eaa8756458ca0cda594(size=11.7 K), total size for store is 23.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:46,244 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:46,244 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/A, priority=13, startTime=1681397625464; duration=0sec 2023-04-13 14:53:46,244 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:46,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:46,247 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:46,247 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:53:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:46,250 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47694 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:53:46,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,250 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/C is initiating minor compaction (all files) 2023-04-13 14:53:46,251 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/C in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:46,251 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/1f44c6dacf3e428190873301edc09497, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/e3fa0d10b5564bcebe6a5efe813a2d61, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/02e5f6ec55be4fbbb7bd29d7dc65f19f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/bb9ba5c0e5d3498393d0095f3d2cf95a] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=46.6 K 2023-04-13 14:53:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:46,251 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 1f44c6dacf3e428190873301edc09497, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681397622018 2023-04-13 14:53:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:46,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:46,260 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/B of 0d965d81bccf303fd24a24be9d6cbc05 into 50d0691c997a41f7a25f7b7df81245af(size=11.7 K), total size for store is 23.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:46,260 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:46,260 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/B, priority=13, startTime=1681397625469; duration=0sec 2023-04-13 14:53:46,260 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:46,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:46,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:46,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:46,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:46,268 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:46,271 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:46,271 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:46,271 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:46,271 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:46,271 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:46,271 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:46,271 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:46,271 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:46,271 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,271 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:46,271 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:46,277 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting e3fa0d10b5564bcebe6a5efe813a2d61, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1681397622480 2023-04-13 14:53:46,278 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 02e5f6ec55be4fbbb7bd29d7dc65f19f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681397623585 2023-04-13 14:53:46,278 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/847a268c61fb444793876829a81e6525 2023-04-13 14:53:46,281 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,281 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,281 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting bb9ba5c0e5d3498393d0095f3d2cf95a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681397624492 2023-04-13 14:53:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:46,282 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,282 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:46,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:46,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13b912eb[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:46,287 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:46,287 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,291 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,291 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:46,291 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:46,291 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:46,292 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,292 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:46,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:46,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:46,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:46,295 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:46,300 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:46,300 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:46,300 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:46,300 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:46,300 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:46,304 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,304 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:46,304 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:46,305 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:46,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:46,307 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:46,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397626211, max timestamp=1681397626298Num uniques 859; 2023-04-13 14:53:46,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397626211, max timestamp=1681397626298Num uniques 859; 2023-04-13 14:53:46,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397626211, max timestamp=1681397626297Num uniques 859; 2023-04-13 14:53:46,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:46,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:46,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,317 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#C#compaction#17 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:46,320 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:46,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:46,320 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:46,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:46,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:46,321 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,321 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:46,321 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:46,321 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,321 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,321 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:46,321 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,321 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:46,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:46,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:46,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:46,329 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:46,329 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:46,348 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/f8b96ee794644b6b849a9ffc3aff20e9 2023-04-13 14:53:46,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/6defd5a7f3e24303baba8b05a414cc9c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6defd5a7f3e24303baba8b05a414cc9c 2023-04-13 14:53:46,364 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/db91fd853c184bf5941540e94a50b945 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/db91fd853c184bf5941540e94a50b945 2023-04-13 14:53:46,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6defd5a7f3e24303baba8b05a414cc9c, entries=200, sequenceid=157, filesize=14.1 K 2023-04-13 14:53:46,381 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/847a268c61fb444793876829a81e6525 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/847a268c61fb444793876829a81e6525 2023-04-13 14:53:46,386 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/C of 0d965d81bccf303fd24a24be9d6cbc05 into db91fd853c184bf5941540e94a50b945(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:46,386 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:46,386 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/C, priority=12, startTime=1681397625469; duration=0sec 2023-04-13 14:53:46,386 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:46,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:46,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397686381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:46,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397686382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,401 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/847a268c61fb444793876829a81e6525, entries=150, sequenceid=157, filesize=11.8 K 2023-04-13 14:53:46,408 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/f8b96ee794644b6b849a9ffc3aff20e9 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/f8b96ee794644b6b849a9ffc3aff20e9 2023-04-13 14:53:46,418 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/f8b96ee794644b6b849a9ffc3aff20e9, entries=150, sequenceid=157, filesize=11.8 K 2023-04-13 14:53:46,421 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 0d965d81bccf303fd24a24be9d6cbc05 in 348ms, sequenceid=157, compaction requested=true 2023-04-13 14:53:46,421 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:46,422 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:46,422 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:46,422 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:53:46,422 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:53:46,422 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:46,424 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38451 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:53:46,424 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/A is initiating minor compaction (all files) 2023-04-13 14:53:46,424 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36061 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:53:46,424 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/A in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:46,424 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/B is initiating minor compaction (all files) 2023-04-13 14:53:46,424 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/B in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:46,424 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/4ec279e6356f4eaa8756458ca0cda594, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/e4ab0937a1ea46e7a68475ac0f0e9193, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6defd5a7f3e24303baba8b05a414cc9c] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=37.5 K 2023-04-13 14:53:46,424 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/50d0691c997a41f7a25f7b7df81245af, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/eb308aa91da84aa79329c8c9627cbbdd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/847a268c61fb444793876829a81e6525] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=35.2 K 2023-04-13 14:53:46,425 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 4ec279e6356f4eaa8756458ca0cda594, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681397623585 2023-04-13 14:53:46,426 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 50d0691c997a41f7a25f7b7df81245af, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681397623585 2023-04-13 14:53:46,427 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting e4ab0937a1ea46e7a68475ac0f0e9193, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681397624492 2023-04-13 14:53:46,428 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting eb308aa91da84aa79329c8c9627cbbdd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681397624492 2023-04-13 14:53:46,428 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 6defd5a7f3e24303baba8b05a414cc9c, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681397625705 2023-04-13 14:53:46,428 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 847a268c61fb444793876829a81e6525, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681397625705 2023-04-13 14:53:46,429 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:46,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:46,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:46,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:46,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:46,448 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:46,448 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:46,453 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:46,453 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:46,454 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,455 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:46,456 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#B#compaction#19 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:46,463 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,464 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:46,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:46,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:46,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:46,464 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,465 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#A#compaction#18 average throughput is 0.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:46,466 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:46,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:46,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:46,470 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:46,470 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:46,472 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:46,473 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:46,473 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:46,473 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:46,473 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:46,473 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-13 14:53:46,473 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:46,473 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:53:46,473 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:46,473 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:53:46,474 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:46,474 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:53:46,506 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:46,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:46,531 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/3a284c2ac6184cfcac875af8ef0b793d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/3a284c2ac6184cfcac875af8ef0b793d 2023-04-13 14:53:46,540 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/ce1aea1510ae4af7b00ecdfc83eb32f4 2023-04-13 14:53:46,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:46,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397686540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:46,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397686542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,549 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/B of 0d965d81bccf303fd24a24be9d6cbc05 into 3a284c2ac6184cfcac875af8ef0b793d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:46,549 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:46,549 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/B, priority=13, startTime=1681397626422; duration=0sec 2023-04-13 14:53:46,549 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:46,549 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-13 14:53:46,552 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 14:53:46,552 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 14:53:46,552 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. because compaction request was cancelled 2023-04-13 14:53:46,596 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/128fab1f7f7a4d4ea0af089f9246eb84 2023-04-13 14:53:46,600 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-13 14:53:46,601 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-13 14:53:46,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:46,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:46,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397686651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397686651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:46,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44146 deadline: 1681397686686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,689 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4194 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:53:46,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:46,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44168 deadline: 1681397686694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,698 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4201 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:53:46,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:46,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44160 deadline: 1681397686726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,730 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4232 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:53:46,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:46,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397686856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:46,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397686856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:46,943 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/95d73b0890d249e985680135b441a8fc as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/95d73b0890d249e985680135b441a8fc 2023-04-13 14:53:46,952 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/A of 0d965d81bccf303fd24a24be9d6cbc05 into 95d73b0890d249e985680135b441a8fc(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:46,953 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:46,953 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/A, priority=13, startTime=1681397626422; duration=0sec 2023-04-13 14:53:46,953 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:47,042 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/4b6348229394409b86553e719b244427 2023-04-13 14:53:47,051 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/ce1aea1510ae4af7b00ecdfc83eb32f4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/ce1aea1510ae4af7b00ecdfc83eb32f4 2023-04-13 14:53:47,061 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/ce1aea1510ae4af7b00ecdfc83eb32f4, entries=150, sequenceid=196, filesize=11.8 K 2023-04-13 14:53:47,065 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/128fab1f7f7a4d4ea0af089f9246eb84 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/128fab1f7f7a4d4ea0af089f9246eb84 2023-04-13 14:53:47,075 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/128fab1f7f7a4d4ea0af089f9246eb84, entries=150, sequenceid=196, filesize=11.8 K 2023-04-13 14:53:47,080 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/4b6348229394409b86553e719b244427 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4b6348229394409b86553e719b244427 2023-04-13 14:53:47,092 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4b6348229394409b86553e719b244427, entries=150, sequenceid=196, filesize=11.8 K 2023-04-13 14:53:47,093 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 0d965d81bccf303fd24a24be9d6cbc05 in 620ms, sequenceid=196, compaction requested=true 2023-04-13 14:53:47,095 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:47,095 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:47,095 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:47,096 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:47,096 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,096 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:47,096 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:47,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,100 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:47,101 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,101 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:47,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:47,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@167d6d97[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:47,103 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:47,103 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,106 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,106 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:47,106 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:47,106 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:47,107 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,107 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:47,108 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:47,108 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:47,108 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:47,113 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:47,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:47,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:47,121 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:47,121 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:47,121 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:47,121 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:47,121 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:47,124 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,124 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:47,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:47,124 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:47,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:47,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:47,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:47,126 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:47,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:47,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,128 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:47,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,131 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,131 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:47,131 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:47,131 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:47,131 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:47,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:47,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:47,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:47,134 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:47,135 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:47,135 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,135 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:47,135 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:47,135 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:47,135 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:47,162 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 14:53:47,169 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:47,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:47,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:47,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:47,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:47,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:47,228 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/0a004fcfcdd34fcf8f4837dadab9de01 2023-04-13 14:53:47,235 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:47,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:47,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:47,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:47,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397687238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:47,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397687248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:47,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:47,256 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:47,256 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:47,269 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:47,269 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:47,270 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,270 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:47,272 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:47,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:47,272 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:47,275 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,275 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:47,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:47,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:47,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:47,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:47,278 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:47,278 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:47,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:47,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:47,301 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:47,311 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:47,311 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:47,311 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:47,311 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:47,311 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:47,311 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,312 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:47,312 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:47,315 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,315 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:47,316 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,316 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:47,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:47,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6727dfa4[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:47,319 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:47,319 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,324 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,324 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:47,324 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:47,324 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:47,324 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,324 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:47,326 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:47,326 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,326 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:47,326 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,347 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/a09184d92f354fef9d1e278613e21ef3 2023-04-13 14:53:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:47,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:47,348 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:47,348 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:47,348 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:47,348 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:47,348 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:47,352 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,353 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:47,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,353 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:47,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:47,353 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:47,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:47,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397687351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397687350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:47,357 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:47,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:47,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,361 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,361 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:47,361 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:47,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:47,361 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:47,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:47,362 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:47,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:47,365 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:47,366 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,366 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:47,366 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,366 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:47,366 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:47,366 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:47,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:47,425 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/74f4ab8a4ab34301a6b629aae9f1f1fa 2023-04-13 14:53:47,436 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/0a004fcfcdd34fcf8f4837dadab9de01 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0a004fcfcdd34fcf8f4837dadab9de01 2023-04-13 14:53:47,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0a004fcfcdd34fcf8f4837dadab9de01, entries=150, sequenceid=211, filesize=11.8 K 2023-04-13 14:53:47,452 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/a09184d92f354fef9d1e278613e21ef3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/a09184d92f354fef9d1e278613e21ef3 2023-04-13 14:53:47,462 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/a09184d92f354fef9d1e278613e21ef3, entries=100, sequenceid=211, filesize=9.4 K 2023-04-13 14:53:47,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/74f4ab8a4ab34301a6b629aae9f1f1fa as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/74f4ab8a4ab34301a6b629aae9f1f1fa 2023-04-13 14:53:47,466 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:47,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:47,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:47,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/74f4ab8a4ab34301a6b629aae9f1f1fa, entries=100, sequenceid=211, filesize=9.4 K 2023-04-13 14:53:47,482 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 0d965d81bccf303fd24a24be9d6cbc05 in 319ms, sequenceid=211, compaction requested=true 2023-04-13 14:53:47,482 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:47,484 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:47,484 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:53:47,484 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:53:47,484 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:47,485 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:47,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:47,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:47,486 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:47,486 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:47,488 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:47,488 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:47,488 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,488 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:47,490 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:47,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:47,490 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:47,491 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,491 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:47,491 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36310 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:53:47,491 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/A is initiating minor compaction (all files) 2023-04-13 14:53:47,491 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/A in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:47,491 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/95d73b0890d249e985680135b441a8fc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/ce1aea1510ae4af7b00ecdfc83eb32f4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0a004fcfcdd34fcf8f4837dadab9de01] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=35.5 K 2023-04-13 14:53:47,492 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:53:47,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:47,493 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 95d73b0890d249e985680135b441a8fc, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681397625705 2023-04-13 14:53:47,492 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/B is initiating minor compaction (all files) 2023-04-13 14:53:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:47,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:47,494 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting ce1aea1510ae4af7b00ecdfc83eb32f4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681397626370 2023-04-13 14:53:47,493 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/B in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:47,494 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/3a284c2ac6184cfcac875af8ef0b793d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/128fab1f7f7a4d4ea0af089f9246eb84, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/a09184d92f354fef9d1e278613e21ef3] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=33.1 K 2023-04-13 14:53:47,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:47,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:47,497 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:47,498 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 3a284c2ac6184cfcac875af8ef0b793d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681397625705 2023-04-13 14:53:47,498 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 0a004fcfcdd34fcf8f4837dadab9de01, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681397626533 2023-04-13 14:53:47,498 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:47,499 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:47,500 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 128fab1f7f7a4d4ea0af089f9246eb84, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681397626370 2023-04-13 14:53:47,508 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:47,508 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:47,509 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:47,509 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:47,509 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 14:53:47,509 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting a09184d92f354fef9d1e278613e21ef3, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681397627160 2023-04-13 14:53:47,509 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:47,510 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:47,510 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:47,510 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:47,510 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:47,510 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:47,545 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#A#compaction#27 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:47,556 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#B#compaction#28 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:47,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:47,577 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:47,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:47,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397687604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:47,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397687615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,668 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/2a98306c02c64fb88ef57dd3c99dc668 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/2a98306c02c64fb88ef57dd3c99dc668 2023-04-13 14:53:47,678 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/A of 0d965d81bccf303fd24a24be9d6cbc05 into 2a98306c02c64fb88ef57dd3c99dc668(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:47,678 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:47,678 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/A, priority=13, startTime=1681397627484; duration=0sec 2023-04-13 14:53:47,679 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:47,679 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:53:47,683 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45888 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:53:47,683 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/C is initiating minor compaction (all files) 2023-04-13 14:53:47,684 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/C in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:47,684 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/db91fd853c184bf5941540e94a50b945, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/f8b96ee794644b6b849a9ffc3aff20e9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4b6348229394409b86553e719b244427, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/74f4ab8a4ab34301a6b629aae9f1f1fa] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=44.8 K 2023-04-13 14:53:47,688 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting db91fd853c184bf5941540e94a50b945, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681397624492 2023-04-13 14:53:47,689 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting f8b96ee794644b6b849a9ffc3aff20e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681397625705 2023-04-13 14:53:47,690 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 4b6348229394409b86553e719b244427, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681397626370 2023-04-13 14:53:47,691 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 74f4ab8a4ab34301a6b629aae9f1f1fa, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681397627160 2023-04-13 14:53:47,716 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#C#compaction#29 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:47,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:47,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:47,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397687732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397687736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:47,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397687943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:47,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397687943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:47,980 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/03bf21dbbe8642ea8b667cd2a405d944 2023-04-13 14:53:48,032 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/7ad98f9c12204d47863189d74e108668 2023-04-13 14:53:48,085 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/aec690d81f0941dcbfcea99d70d17a08 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/aec690d81f0941dcbfcea99d70d17a08 2023-04-13 14:53:48,100 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/B of 0d965d81bccf303fd24a24be9d6cbc05 into aec690d81f0941dcbfcea99d70d17a08(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:48,101 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:48,101 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/B, priority=13, startTime=1681397627484; duration=0sec 2023-04-13 14:53:48,101 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:48,171 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/4eaf886ea1784a509e6abcf67e9f802e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4eaf886ea1784a509e6abcf67e9f802e 2023-04-13 14:53:48,181 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/C of 0d965d81bccf303fd24a24be9d6cbc05 into 4eaf886ea1784a509e6abcf67e9f802e(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:48,181 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:48,181 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/C, priority=12, startTime=1681397627485; duration=0sec 2023-04-13 14:53:48,181 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:48,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:48,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44182 deadline: 1681397688247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:48,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44164 deadline: 1681397688248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,518 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/ef80b0a5c3d54e12bd9d6e399ec38dae 2023-04-13 14:53:48,541 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/03bf21dbbe8642ea8b667cd2a405d944 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/03bf21dbbe8642ea8b667cd2a405d944 2023-04-13 14:53:48,552 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/03bf21dbbe8642ea8b667cd2a405d944, entries=150, sequenceid=230, filesize=11.8 K 2023-04-13 14:53:48,554 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/7ad98f9c12204d47863189d74e108668 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/7ad98f9c12204d47863189d74e108668 2023-04-13 14:53:48,573 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/7ad98f9c12204d47863189d74e108668, entries=150, sequenceid=230, filesize=11.8 K 2023-04-13 14:53:48,576 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/ef80b0a5c3d54e12bd9d6e399ec38dae as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/ef80b0a5c3d54e12bd9d6e399ec38dae 2023-04-13 14:53:48,586 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/ef80b0a5c3d54e12bd9d6e399ec38dae, entries=150, sequenceid=230, filesize=11.8 K 2023-04-13 14:53:48,588 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 0d965d81bccf303fd24a24be9d6cbc05 in 1079ms, sequenceid=230, compaction requested=false 2023-04-13 14:53:48,589 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:48,589 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:48,589 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:48,599 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:48,599 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,599 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:48,599 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:48,604 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,604 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:48,604 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,604 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23e90e9e[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:48,607 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:48,607 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,609 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,609 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:48,609 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:48,609 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:48,610 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,610 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:48,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,612 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:48,612 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:48,612 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:48,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:48,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:48,619 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:48,619 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:48,619 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:48,619 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:48,619 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:48,621 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,621 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,621 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:48,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:48,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:48,622 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,624 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,632 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:48,632 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,632 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:48,632 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:48,632 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:48,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:48,632 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:48,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:48,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:48,632 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:48,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:48,638 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:48,638 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:48,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,739 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:48,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:48,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:48,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:48,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:48,744 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:48,744 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:48,749 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:48,749 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:48,749 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,749 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:48,751 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,751 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:48,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:48,751 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,751 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:48,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:48,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:48,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:48,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:48,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:48,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:48,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:48,755 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:48,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:48,755 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 14:53:48,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:48,756 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:48,756 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:48,756 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:48,756 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:48,756 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:48,757 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:48,757 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:48,757 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:48,757 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:48,760 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:48,761 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,763 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,763 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:48,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:48,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@781c7e42[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:48,770 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:48,770 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,773 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,773 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,774 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:48,774 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:48,774 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:48,774 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:48,777 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:48,778 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:48,778 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:48,778 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:48,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:48,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:48,793 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:48,793 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:48,793 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:48,793 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:48,793 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:48,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:48,798 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397688792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,797 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,797 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:48,799 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:48,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:48,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:48,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:48,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397688796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:48,802 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:48,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,805 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:48,805 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:48,805 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,805 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:48,805 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:48,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:48,805 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:48,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:48,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:48,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:48,806 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:48,806 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:48,807 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:48,806 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,807 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,807 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,850 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/70c6b49fead049f89996eb40d3886f8e 2023-04-13 14:53:48,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:48,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397688902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:48,907 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:48,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397688903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:48,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:48,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:48,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:48,918 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:48,918 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:48,920 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:48,921 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:48,921 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,921 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:48,922 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,923 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:48,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:48,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:48,923 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,923 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:48,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:48,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:48,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:48,934 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:48,934 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:48,935 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:48,936 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:48,936 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:48,936 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:48,936 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:48,936 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:48,936 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,936 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:48,936 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:48,944 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,944 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:48,945 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,945 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:48,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:48,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f0b503[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:48,947 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:48,947 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,948 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:48,949 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,949 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:48,951 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:48,951 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,951 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:48,951 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:48,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:48,956 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:48,956 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:48,956 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:48,956 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:48,956 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:48,961 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,961 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,961 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:48,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:48,963 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,965 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:48,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:48,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:48,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:48,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:48,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:48,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:48,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:48,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:48,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:48,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:48,970 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:48,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:48,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:48,970 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:48,970 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:49,071 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:49,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:49,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:49,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:49,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:49,082 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:49,082 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:49,083 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:49,083 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:49,084 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,084 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,086 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:49,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:49,086 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,086 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,086 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:49,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:49,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:49,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:49,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:49,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:49,089 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:49,089 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:49,089 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:49,089 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:49,089 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:49,090 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:49,090 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:49,090 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:49,090 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:49,090 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,090 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:49,090 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:49,092 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,092 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:49,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:49,093 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,093 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:49,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:49,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:49,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:49,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:49,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c7380e6[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:49,095 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:49,095 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,096 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:49,096 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,096 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:49,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:49,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:49,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:49,101 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:49,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:49,108 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:49,108 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:49,108 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:49,108 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:49,108 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:49,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:49,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:49,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:49,112 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,114 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:49,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:49,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:49,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397689109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397689110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,120 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,120 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,120 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,120 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,120 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,120 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,120 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,120 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:49,120 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:49,120 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:49,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:49,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:49,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:49,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:49,124 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:49,124 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:49,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:49,224 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:49,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:49,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:49,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:49,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:49,230 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:49,230 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:49,231 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:49,231 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:49,232 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,232 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,233 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:49,233 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:49,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:49,233 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,233 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:49,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:49,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:49,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:49,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:49,235 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:49,235 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:49,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:49,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:49,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:49,238 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:49,238 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:49,238 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:49,238 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:49,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:49,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:49,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:49,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:49,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,240 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:49,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:49,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:49,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3705f808[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:49,243 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:49,243 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,244 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:49,245 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,245 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:49,247 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:49,247 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:49,247 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:49,248 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:49,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:49,253 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:49,253 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:49,253 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:49,253 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:49,253 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:49,255 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,255 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:49,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,255 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:49,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:49,255 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,257 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:49,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:49,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:49,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:49,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:49,266 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:49,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:49,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,266 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:49,266 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:49,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,267 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,269 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:49,285 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/6541661e553c4cbb8adaf9e12389d231 2023-04-13 14:53:49,366 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:49,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:49,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:49,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:49,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:49,393 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:49,393 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:49,393 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:49,393 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:49,393 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,393 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,395 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:49,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,397 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,397 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:49,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:49,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:49,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:49,398 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:49,398 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:49,398 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:49,398 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:49,401 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,401 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:49,403 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,403 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:49,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:49,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c707ec4[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:49,405 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:49,405 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,406 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,406 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:49,406 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:49,406 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:49,406 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,406 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:49,408 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,408 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:49,408 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:49,408 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:49,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:49,411 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:49,411 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:49,411 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:49,411 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:49,411 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:49,413 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,413 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:49,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,413 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:49,413 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:49,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397689418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:49,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397689418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,420 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,420 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,420 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:49,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:49,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:49,421 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:49,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:49,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:49,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:49,421 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:49,421 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:49,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:49,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:49,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:49,521 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:49,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:49,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:49,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:49,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:49,525 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:49,525 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:49,526 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:49,526 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:49,526 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,526 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,527 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,527 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:49,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:49,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:49,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:49,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:49,530 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:49,530 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:49,530 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:49,530 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:49,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:49,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:49,533 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,533 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:49,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:49,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:49,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c289ee4[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:49,535 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:49,535 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,537 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,538 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:49,538 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:49,538 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:49,538 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,538 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:49,540 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:49,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:49,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:49,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:49,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:49,545 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:49,545 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:49,545 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:49,545 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:49,545 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:49,553 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:49,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:49,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:49,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:49,556 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,558 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,560 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:49,560 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:49,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:49,560 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,561 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,561 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,561 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:49,561 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,561 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,561 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:49,564 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:49,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:49,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:49,566 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:49,566 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:49,667 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:49,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:49,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:49,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:49,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:49,677 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:49,677 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:49,678 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:49,678 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:49,680 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,680 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,681 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:49,682 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:49,682 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,682 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:49,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:49,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:49,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:49,687 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:49,687 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:49,687 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:49,688 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:49,688 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:49,688 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:49,688 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:49,688 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:49,688 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:49,688 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:49,688 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:49,688 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,688 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:49,688 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:49,691 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,691 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:49,691 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,691 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:49,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:49,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c1c4f37[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:49,694 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:49,694 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,695 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,695 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:49,695 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:49,695 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:49,696 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,697 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:49,697 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:49,697 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:49,697 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:49,697 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:49,697 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:49,697 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:49,699 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,699 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:49,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,699 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:49,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:49,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,702 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:49,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:49,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:49,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:49,707 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:49,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:49,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:49,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:49,707 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:49,707 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:49,715 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/08e460facade4839b31085498779b228 2023-04-13 14:53:49,726 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/70c6b49fead049f89996eb40d3886f8e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/70c6b49fead049f89996eb40d3886f8e 2023-04-13 14:53:49,734 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/70c6b49fead049f89996eb40d3886f8e, entries=100, sequenceid=249, filesize=9.4 K 2023-04-13 14:53:49,737 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/6541661e553c4cbb8adaf9e12389d231 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/6541661e553c4cbb8adaf9e12389d231 2023-04-13 14:53:49,749 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/6541661e553c4cbb8adaf9e12389d231, entries=100, sequenceid=249, filesize=9.4 K 2023-04-13 14:53:49,752 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/08e460facade4839b31085498779b228 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/08e460facade4839b31085498779b228 2023-04-13 14:53:49,759 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/08e460facade4839b31085498779b228, entries=100, sequenceid=249, filesize=9.4 K 2023-04-13 14:53:49,763 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 0d965d81bccf303fd24a24be9d6cbc05 in 1008ms, sequenceid=249, compaction requested=true 2023-04-13 14:53:49,763 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:49,763 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:49,763 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:53:49,763 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:53:49,763 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:49,764 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:49,769 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:53:49,769 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/A is initiating minor compaction (all files) 2023-04-13 14:53:49,769 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:53:49,769 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/A in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:49,769 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/B is initiating minor compaction (all files) 2023-04-13 14:53:49,769 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/B in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:49,769 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/2a98306c02c64fb88ef57dd3c99dc668, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/03bf21dbbe8642ea8b667cd2a405d944, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/70c6b49fead049f89996eb40d3886f8e] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=33.2 K 2023-04-13 14:53:49,770 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/aec690d81f0941dcbfcea99d70d17a08, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/7ad98f9c12204d47863189d74e108668, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/6541661e553c4cbb8adaf9e12389d231] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=33.2 K 2023-04-13 14:53:49,770 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 2a98306c02c64fb88ef57dd3c99dc668, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681397626370 2023-04-13 14:53:49,771 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting aec690d81f0941dcbfcea99d70d17a08, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681397626370 2023-04-13 14:53:49,772 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 03bf21dbbe8642ea8b667cd2a405d944, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1681397627214 2023-04-13 14:53:49,772 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 7ad98f9c12204d47863189d74e108668, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1681397627214 2023-04-13 14:53:49,773 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 70c6b49fead049f89996eb40d3886f8e, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681397628752 2023-04-13 14:53:49,773 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 6541661e553c4cbb8adaf9e12389d231, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681397628752 2023-04-13 14:53:49,791 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#A#compaction#36 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:49,792 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#B#compaction#35 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:49,807 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:49,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:49,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:49,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:49,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:49,812 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:49,812 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:49,812 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:49,812 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:49,813 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,813 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,814 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:49,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:49,814 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,815 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,815 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:49,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:49,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:49,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:49,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:49,816 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:49,816 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:49,816 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/07136225dadd4b63b55a18395d1331a2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/07136225dadd4b63b55a18395d1331a2 2023-04-13 14:53:49,816 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:49,817 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:49,817 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:49,817 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:49,817 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:49,818 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 14:53:49,818 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:49,818 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:49,818 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:49,818 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:49,818 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:49,818 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:49,827 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/A of 0d965d81bccf303fd24a24be9d6cbc05 into 07136225dadd4b63b55a18395d1331a2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:49,827 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:49,827 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/A, priority=13, startTime=1681397629763; duration=0sec 2023-04-13 14:53:49,827 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:49,827 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:53:49,829 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:53:49,830 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/C is initiating minor compaction (all files) 2023-04-13 14:53:49,830 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/C in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:49,830 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4eaf886ea1784a509e6abcf67e9f802e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/ef80b0a5c3d54e12bd9d6e399ec38dae, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/08e460facade4839b31085498779b228] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=33.2 K 2023-04-13 14:53:49,830 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 4eaf886ea1784a509e6abcf67e9f802e, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681397626370 2023-04-13 14:53:49,832 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting ef80b0a5c3d54e12bd9d6e399ec38dae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1681397627214 2023-04-13 14:53:49,833 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 08e460facade4839b31085498779b228, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681397628752 2023-04-13 14:53:49,845 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#C#compaction#38 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:49,847 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/b2bfd462b03f41d594f135cbbc56e8b1 2023-04-13 14:53:49,882 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/f399207c8c7240ad8c1ced4d1f7ba298 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/f399207c8c7240ad8c1ced4d1f7ba298 2023-04-13 14:53:49,890 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/C of 0d965d81bccf303fd24a24be9d6cbc05 into f399207c8c7240ad8c1ced4d1f7ba298(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:49,891 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:49,891 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/C, priority=13, startTime=1681397629764; duration=0sec 2023-04-13 14:53:49,891 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:49,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:49,925 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:49,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:49,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397689961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:49,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:49,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397689962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:50,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397690066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:50,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397690066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,217 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/a35837b696e34e17ab990269d34505fe as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/a35837b696e34e17ab990269d34505fe 2023-04-13 14:53:50,237 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/B of 0d965d81bccf303fd24a24be9d6cbc05 into a35837b696e34e17ab990269d34505fe(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:50,237 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:50,237 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/B, priority=13, startTime=1681397629763; duration=0sec 2023-04-13 14:53:50,237 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:50,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:50,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397690270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397690270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,293 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/e6422ef74ca341d99a6f245a104228ea 2023-04-13 14:53:50,331 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/a4d1e13a40504fdfa896ae6b1dab4f60 2023-04-13 14:53:50,345 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/b2bfd462b03f41d594f135cbbc56e8b1 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/b2bfd462b03f41d594f135cbbc56e8b1 2023-04-13 14:53:50,352 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/b2bfd462b03f41d594f135cbbc56e8b1, entries=150, sequenceid=265, filesize=11.9 K 2023-04-13 14:53:50,353 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/e6422ef74ca341d99a6f245a104228ea as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/e6422ef74ca341d99a6f245a104228ea 2023-04-13 14:53:50,369 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/e6422ef74ca341d99a6f245a104228ea, entries=150, sequenceid=265, filesize=11.9 K 2023-04-13 14:53:50,371 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/a4d1e13a40504fdfa896ae6b1dab4f60 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/a4d1e13a40504fdfa896ae6b1dab4f60 2023-04-13 14:53:50,379 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/a4d1e13a40504fdfa896ae6b1dab4f60, entries=150, sequenceid=265, filesize=11.9 K 2023-04-13 14:53:50,380 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 0d965d81bccf303fd24a24be9d6cbc05 in 563ms, sequenceid=265, compaction requested=false 2023-04-13 14:53:50,380 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:50,380 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:50,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:50,382 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:50,382 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,382 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:50,382 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:50,384 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:50,385 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:50,385 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:50,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:50,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74c12f04[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:50,390 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:50,390 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:50,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:50,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:50,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:50,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:50,392 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,392 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:50,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:50,394 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:50,394 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,394 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:50,394 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:50,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:50,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:50,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:50,399 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:50,399 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:50,399 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:50,399 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:50,399 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:50,401 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:50,401 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:50,401 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:50,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:50,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:50,410 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,412 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,418 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,419 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:50,419 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:50,419 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:50,419 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:50,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:50,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:50,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:50,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:50,422 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:50,422 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:50,422 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:50,422 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:50,422 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:50,422 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:50,522 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:50,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:50,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:50,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:50,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:50,528 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:50,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:50,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:50,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:50,529 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:50,529 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:50,531 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:50,531 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:50,532 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,532 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:50,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:50,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:50,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:50,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:50,536 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:50,536 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:50,536 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:50,536 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:50,536 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:50,536 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:50,536 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:50,537 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 14:53:50,537 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:50,537 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:50,538 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:50,538 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:50,538 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:50,538 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:50,562 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/3466d1246f294bbb86d8a1a56ed321bb 2023-04-13 14:53:50,579 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:50,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:50,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:50,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397690598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:50,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397690599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:50,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397690702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:50,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397690704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:50,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44168 deadline: 1681397690705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,712 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8211 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:53:50,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:50,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44146 deadline: 1681397690713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,717 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8221 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:53:50,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:50,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44160 deadline: 1681397690736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,741 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8243 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:53:50,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:50,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397690906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:50,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397690908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:50,979 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/49170255b8c3485fb51c313bf125147e 2023-04-13 14:53:51,002 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/4886b370c79b4d76868dc8e6e6b6d0fe 2023-04-13 14:53:51,011 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/3466d1246f294bbb86d8a1a56ed321bb as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/3466d1246f294bbb86d8a1a56ed321bb 2023-04-13 14:53:51,020 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/3466d1246f294bbb86d8a1a56ed321bb, entries=150, sequenceid=284, filesize=11.9 K 2023-04-13 14:53:51,022 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/49170255b8c3485fb51c313bf125147e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/49170255b8c3485fb51c313bf125147e 2023-04-13 14:53:51,030 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/49170255b8c3485fb51c313bf125147e, entries=150, sequenceid=284, filesize=11.9 K 2023-04-13 14:53:51,034 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/4886b370c79b4d76868dc8e6e6b6d0fe as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4886b370c79b4d76868dc8e6e6b6d0fe 2023-04-13 14:53:51,043 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4886b370c79b4d76868dc8e6e6b6d0fe, entries=150, sequenceid=284, filesize=11.9 K 2023-04-13 14:53:51,044 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 0d965d81bccf303fd24a24be9d6cbc05 in 507ms, sequenceid=284, compaction requested=true 2023-04-13 14:53:51,044 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:51,044 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:51,045 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:51,046 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:51,046 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,046 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:51,046 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:51,048 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,048 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:51,048 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:51,048 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:51,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:51,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b68e906[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:51,061 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:51,061 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:51,064 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:51,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:51,064 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,064 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:51,066 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:51,066 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:51,066 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:51,066 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:51,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:51,070 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:51,070 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:51,070 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:51,070 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:51,071 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:51,072 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:51,073 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:51,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:51,073 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:51,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:51,074 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:51,085 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:51,088 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,088 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:51,088 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:51,088 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,088 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:51,088 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:51,088 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:51,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:51,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:51,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:51,088 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:51,088 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:51,088 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:51,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:51,089 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:51,090 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:51,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:51,190 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:51,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:51,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:51,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:51,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:51,196 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:51,196 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:51,196 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:51,196 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:51,202 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:51,202 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:51,204 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:51,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:51,204 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,205 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,205 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:51,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:51,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:51,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:51,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:51,207 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:51,207 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:51,207 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:51,207 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:51,207 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:51,207 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:51,207 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:51,207 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 14:53:51,207 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:51,208 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:51,208 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:51,208 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:51,208 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:51,208 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:51,215 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:51,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:51,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:51,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397691242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:51,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397691244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:51,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397691346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:51,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397691347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:51,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397691550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:51,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397691551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,653 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/d2b4cde328b945caa371a85a61b92778 2023-04-13 14:53:51,691 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/06411e287bde42428169cce4bf571e3f 2023-04-13 14:53:51,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:51,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397691854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:51,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:51,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397691855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,105 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/9db0b6253a5f45eb8068842c038dfdf5 2023-04-13 14:53:52,112 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/d2b4cde328b945caa371a85a61b92778 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/d2b4cde328b945caa371a85a61b92778 2023-04-13 14:53:52,119 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/d2b4cde328b945caa371a85a61b92778, entries=150, sequenceid=300, filesize=11.9 K 2023-04-13 14:53:52,119 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/06411e287bde42428169cce4bf571e3f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/06411e287bde42428169cce4bf571e3f 2023-04-13 14:53:52,125 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/06411e287bde42428169cce4bf571e3f, entries=150, sequenceid=300, filesize=11.9 K 2023-04-13 14:53:52,126 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/9db0b6253a5f45eb8068842c038dfdf5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/9db0b6253a5f45eb8068842c038dfdf5 2023-04-13 14:53:52,131 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/9db0b6253a5f45eb8068842c038dfdf5, entries=150, sequenceid=300, filesize=11.9 K 2023-04-13 14:53:52,132 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 0d965d81bccf303fd24a24be9d6cbc05 in 925ms, sequenceid=300, compaction requested=true 2023-04-13 14:53:52,132 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:52,132 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:52,132 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:52,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,137 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:52,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:52,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:52,137 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:52,137 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:52,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:52,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:52,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@335f9331[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:52,139 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:52,139 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:52,140 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,141 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:52,141 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:52,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:52,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:52,141 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:52,141 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:52,141 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:52,148 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:52,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:52,151 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:52,151 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:52,151 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:52,151 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:52,151 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:52,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:52,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:52,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:52,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:52,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:52,153 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:52,156 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,159 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,160 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:52,160 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:52,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:52,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:52,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:52,160 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:52,160 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:52,160 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:52,160 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,160 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:52,160 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:52,160 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:52,160 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:52,160 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:52,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:52,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:52,260 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:52,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:52,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:52,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:52,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:52,268 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:52,268 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:52,268 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:52,268 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:52,269 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:52,269 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:52,270 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:52,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:52,270 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:52,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:52,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:52,275 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,275 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:52,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:52,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:52,280 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:52,280 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:52,281 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 14:53:52,281 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:52,281 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:52,281 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:52,281 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:52,281 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:52,281 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:52,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:52,360 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:52,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:52,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397692388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:52,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397692390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:52,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397692495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:52,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397692495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:52,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397692704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:52,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397692706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:52,714 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/e932242b99a54841a07ed036a10dd0d5 2023-04-13 14:53:52,731 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/e302f001da414303b03257de81ebdcc2 2023-04-13 14:53:53,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:53,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397693008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:53,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397693011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,143 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/5e0e0ce52a4f4434b3f25fea5f4e1ec0 2023-04-13 14:53:53,150 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/e932242b99a54841a07ed036a10dd0d5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/e932242b99a54841a07ed036a10dd0d5 2023-04-13 14:53:53,155 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/e932242b99a54841a07ed036a10dd0d5, entries=150, sequenceid=316, filesize=11.9 K 2023-04-13 14:53:53,156 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/e302f001da414303b03257de81ebdcc2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/e302f001da414303b03257de81ebdcc2 2023-04-13 14:53:53,161 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/e302f001da414303b03257de81ebdcc2, entries=150, sequenceid=316, filesize=11.9 K 2023-04-13 14:53:53,162 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/5e0e0ce52a4f4434b3f25fea5f4e1ec0 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/5e0e0ce52a4f4434b3f25fea5f4e1ec0 2023-04-13 14:53:53,167 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/5e0e0ce52a4f4434b3f25fea5f4e1ec0, entries=150, sequenceid=316, filesize=11.9 K 2023-04-13 14:53:53,169 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 0d965d81bccf303fd24a24be9d6cbc05 in 887ms, sequenceid=316, compaction requested=true 2023-04-13 14:53:53,169 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:53,169 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:53,169 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:53,169 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:53,169 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,169 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:53,169 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:53,171 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,171 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:53,171 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,171 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:53,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:53,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d34079a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:53,173 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:53,173 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,174 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,174 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:53,175 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,175 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:53,176 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:53,176 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:53,176 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:53,176 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:53,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:53,179 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:53,179 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:53,179 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:53,179 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:53,179 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:53,182 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,182 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:53,182 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:53,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:53,184 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:53,185 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,188 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:53,188 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:53,188 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:53,188 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,188 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,188 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:53,188 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:53,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:53,188 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,188 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,188 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,189 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:53,189 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:53,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:53,289 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:53,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:53,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:53,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:53,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:53,299 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:53,299 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:53,299 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:53,299 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:53,305 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,305 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,306 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:53,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:53,306 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,307 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,307 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:53,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:53,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:53,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:53,308 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:53,308 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:53,308 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:53,308 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:53,309 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:53,311 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:53,311 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:53,311 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 14:53:53,311 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:53,311 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:53,312 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:53,312 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:53,312 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:53,312 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:53,350 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/a6897d249815418c86e2e855c682d286 2023-04-13 14:53:53,385 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/71bb9d05b2dd4730b913b5febb3c9a10 2023-04-13 14:53:53,405 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/2e5ba16bbb914dbca1ce3118080abf77 2023-04-13 14:53:53,413 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/a6897d249815418c86e2e855c682d286 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/a6897d249815418c86e2e855c682d286 2023-04-13 14:53:53,422 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/a6897d249815418c86e2e855c682d286, entries=150, sequenceid=332, filesize=11.9 K 2023-04-13 14:53:53,423 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/71bb9d05b2dd4730b913b5febb3c9a10 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/71bb9d05b2dd4730b913b5febb3c9a10 2023-04-13 14:53:53,433 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/71bb9d05b2dd4730b913b5febb3c9a10, entries=150, sequenceid=332, filesize=11.9 K 2023-04-13 14:53:53,434 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/2e5ba16bbb914dbca1ce3118080abf77 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/2e5ba16bbb914dbca1ce3118080abf77 2023-04-13 14:53:53,441 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/2e5ba16bbb914dbca1ce3118080abf77, entries=150, sequenceid=332, filesize=11.9 K 2023-04-13 14:53:53,442 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 0d965d81bccf303fd24a24be9d6cbc05 in 131ms, sequenceid=332, compaction requested=true 2023-04-13 14:53:53,443 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:53,443 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:53,443 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:53,443 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:53,443 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,443 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:53,443 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:53,449 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,449 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:53,450 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,450 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:53,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:53,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4070b637[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:53,452 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:53,452 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,453 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,454 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:53,454 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:53,454 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:53,454 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,454 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:53,455 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:53,455 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,455 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:53,455 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:53,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:53,458 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:53,458 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:53,458 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:53,458 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:53,458 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:53,459 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,459 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,460 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:53,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:53,461 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,463 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,468 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:53,468 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:53,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:53,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:53,469 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,469 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,469 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,469 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,469 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,469 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:53,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:53,469 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:53,469 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,469 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:53,469 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:53,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:53,531 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:53:53,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:53,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:53,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:53,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:53,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:53,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:53,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/06bb15fd4bdd4c10b32f3733c331a695 2023-04-13 14:53:53,569 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:53,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:53,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:53,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:53,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:53,600 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:53,600 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:53,600 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:53,600 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:53,601 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,601 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,602 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:53,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:53,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397633578, max timestamp=1681397633601Num uniques 859; 2023-04-13 14:53:53,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,603 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397633578, max timestamp=1681397633601Num uniques 859; 2023-04-13 14:53:53,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:53,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397633578, max timestamp=1681397633601Num uniques 859; 2023-04-13 14:53:53,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:53,608 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,608 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:53,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:53,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:53,620 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/3c74c93732544dbb813193e7f66640ec 2023-04-13 14:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:53,621 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:53,621 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:53,622 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:53,622 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:53,622 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:53,622 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:53,622 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:53,622 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:53,622 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,622 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:53,622 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:53,624 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,625 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,625 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ccc57e6[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:53,628 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:53,628 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,630 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,630 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:53,630 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:53,630 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:53,630 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,630 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:53,632 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:53,633 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:53,633 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:53,633 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:53,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:53,635 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:53,635 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:53,635 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:53,635 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:53,635 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:53,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:53,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:53,638 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,639 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,642 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:53,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:53,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:53,643 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:53,643 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:53,647 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:53,647 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:53,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:53,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:53,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:53,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:53,647 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,648 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,648 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,648 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,648 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,648 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,648 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:53,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397693659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:53,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397693660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,743 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:53,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:53,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:53,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:53,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:53,748 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:53,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:53,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:53,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:53,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,750 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:53,750 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:53,750 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,750 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:53,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:53,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:53,752 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:53,752 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:53,752 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:53,752 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:53,758 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:53,758 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:53,758 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,759 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:53,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:53,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13c58155[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:53,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:53,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,762 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,762 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:53,762 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,762 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:53,762 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:53,762 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:53,763 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:53,763 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:53,764 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:53,764 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:53,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:53,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397693762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:53,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:53,767 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:53,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:53,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:53,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:53,767 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:53,769 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,769 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,769 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:53,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:53,770 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,772 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:53,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:53,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397693768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,777 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:53,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:53,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:53,777 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:53,777 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:53,777 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:53,777 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:53,777 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,777 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:53,777 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,777 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,777 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,777 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:53,777 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,877 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:53,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:53,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:53,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:53,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:53,884 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:53,884 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:53,884 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:53,884 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:53,885 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,885 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,886 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:53,886 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:53,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:53,887 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,888 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:53,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:53,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:53,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:53,889 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:53,889 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:53,889 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:53,889 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:53,889 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:53,889 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:53,890 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:53,890 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:53,890 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:53,890 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:53,890 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:53,890 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,890 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:53,890 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:53,891 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,891 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:53,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:53,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:53,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e504c60[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:53,894 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:53,894 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,899 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,899 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,899 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:53,899 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:53,899 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:53,899 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:53,900 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:53,900 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,900 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:53,900 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:53,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:53,903 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:53,903 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:53,903 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:53,903 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:53,903 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:53,905 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,905 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,905 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:53,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:53,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:53,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:53,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,906 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,907 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,911 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:53,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:53,911 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:53,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:53,913 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,913 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,913 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:53,913 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,913 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,913 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:53,913 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:53,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:53,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:53,913 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:53,914 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:53,914 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:53,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:53,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397693968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:53,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:53,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397693974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,014 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:54,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:54,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:54,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:54,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:54,019 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:54,019 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:54,019 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:54,019 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:54,020 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,020 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,021 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,021 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:54,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:54,021 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,021 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:54,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:54,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:54,023 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:54,023 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:54,024 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:54,024 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:54,025 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,025 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,026 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,026 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:54,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bd48293[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:54,027 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:54,027 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,028 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,029 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:54,029 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:54,029 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:54,029 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,029 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:54,031 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,031 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,031 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:54,031 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:54,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:54,034 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:54,034 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:54,034 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:54,034 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:54,034 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:54,036 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,036 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,036 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:54,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,038 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,042 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,045 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,045 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:54,045 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:54,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:54,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:54,045 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,045 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,045 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,045 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,045 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,045 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:54,045 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:54,045 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:54,046 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:54,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:54,047 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/96d18b6afdd84fa6a0dbdd6b43fd3473 2023-04-13 14:53:54,055 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/06bb15fd4bdd4c10b32f3733c331a695 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/06bb15fd4bdd4c10b32f3733c331a695 2023-04-13 14:53:54,061 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/06bb15fd4bdd4c10b32f3733c331a695, entries=200, sequenceid=343, filesize=14.3 K 2023-04-13 14:53:54,062 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/3c74c93732544dbb813193e7f66640ec as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/3c74c93732544dbb813193e7f66640ec 2023-04-13 14:53:54,069 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/3c74c93732544dbb813193e7f66640ec, entries=150, sequenceid=343, filesize=11.9 K 2023-04-13 14:53:54,071 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/96d18b6afdd84fa6a0dbdd6b43fd3473 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/96d18b6afdd84fa6a0dbdd6b43fd3473 2023-04-13 14:53:54,079 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/96d18b6afdd84fa6a0dbdd6b43fd3473, entries=150, sequenceid=343, filesize=11.9 K 2023-04-13 14:53:54,080 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 0d965d81bccf303fd24a24be9d6cbc05 in 549ms, sequenceid=343, compaction requested=true 2023-04-13 14:53:54,080 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:54,081 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:54,081 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 14:53:54,081 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:54,081 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 14:53:54,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:54,084 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 88002 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 14:53:54,084 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/A is initiating minor compaction (all files) 2023-04-13 14:53:54,084 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/A in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:54,084 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/07136225dadd4b63b55a18395d1331a2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/b2bfd462b03f41d594f135cbbc56e8b1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/3466d1246f294bbb86d8a1a56ed321bb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/d2b4cde328b945caa371a85a61b92778, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/e932242b99a54841a07ed036a10dd0d5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/a6897d249815418c86e2e855c682d286, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/06bb15fd4bdd4c10b32f3733c331a695] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=85.9 K 2023-04-13 14:53:54,085 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 07136225dadd4b63b55a18395d1331a2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681397627229 2023-04-13 14:53:54,085 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85562 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 14:53:54,085 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/B is initiating minor compaction (all files) 2023-04-13 14:53:54,085 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/B in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:54,085 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/a35837b696e34e17ab990269d34505fe, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/e6422ef74ca341d99a6f245a104228ea, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/49170255b8c3485fb51c313bf125147e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/06411e287bde42428169cce4bf571e3f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/e302f001da414303b03257de81ebdcc2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/71bb9d05b2dd4730b913b5febb3c9a10, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/3c74c93732544dbb813193e7f66640ec] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=83.6 K 2023-04-13 14:53:54,086 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting b2bfd462b03f41d594f135cbbc56e8b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681397628776 2023-04-13 14:53:54,086 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting a35837b696e34e17ab990269d34505fe, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681397627229 2023-04-13 14:53:54,086 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 3466d1246f294bbb86d8a1a56ed321bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681397629952 2023-04-13 14:53:54,087 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting e6422ef74ca341d99a6f245a104228ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681397628776 2023-04-13 14:53:54,087 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting d2b4cde328b945caa371a85a61b92778, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681397630592 2023-04-13 14:53:54,087 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 49170255b8c3485fb51c313bf125147e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681397629952 2023-04-13 14:53:54,088 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting e932242b99a54841a07ed036a10dd0d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1681397631229 2023-04-13 14:53:54,089 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting a6897d249815418c86e2e855c682d286, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1681397632380 2023-04-13 14:53:54,089 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 06bb15fd4bdd4c10b32f3733c331a695, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681397633518 2023-04-13 14:53:54,090 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 06411e287bde42428169cce4bf571e3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681397630592 2023-04-13 14:53:54,091 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting e302f001da414303b03257de81ebdcc2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1681397631229 2023-04-13 14:53:54,092 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 71bb9d05b2dd4730b913b5febb3c9a10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1681397632380 2023-04-13 14:53:54,092 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 3c74c93732544dbb813193e7f66640ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681397633518 2023-04-13 14:53:54,114 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#A#compaction#56 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:54,116 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#B#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:54,145 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/397a23b4d83d45179c0905a64313c8cf as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/397a23b4d83d45179c0905a64313c8cf 2023-04-13 14:53:54,146 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:54,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:54,148 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/1581635eab2e468eb8f5441064a1cb99 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/1581635eab2e468eb8f5441064a1cb99 2023-04-13 14:53:54,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:54,156 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/A of 0d965d81bccf303fd24a24be9d6cbc05 into 397a23b4d83d45179c0905a64313c8cf(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:54,156 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:54,156 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/A, priority=9, startTime=1681397634081; duration=0sec 2023-04-13 14:53:54,156 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:54,156 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 14:53:54,158 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/B of 0d965d81bccf303fd24a24be9d6cbc05 into 1581635eab2e468eb8f5441064a1cb99(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:54,158 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:54,158 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/B, priority=9, startTime=1681397634081; duration=0sec 2023-04-13 14:53:54,158 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:54,160 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85528 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 14:53:54,160 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/C is initiating minor compaction (all files) 2023-04-13 14:53:54,160 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/C in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:54,160 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/f399207c8c7240ad8c1ced4d1f7ba298, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/a4d1e13a40504fdfa896ae6b1dab4f60, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4886b370c79b4d76868dc8e6e6b6d0fe, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/9db0b6253a5f45eb8068842c038dfdf5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/5e0e0ce52a4f4434b3f25fea5f4e1ec0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/2e5ba16bbb914dbca1ce3118080abf77, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/96d18b6afdd84fa6a0dbdd6b43fd3473] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=83.5 K 2023-04-13 14:53:54,161 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting f399207c8c7240ad8c1ced4d1f7ba298, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681397627229 2023-04-13 14:53:54,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:54,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:54,161 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:54,161 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:54,161 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:54,161 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:54,161 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting a4d1e13a40504fdfa896ae6b1dab4f60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681397628776 2023-04-13 14:53:54,162 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 4886b370c79b4d76868dc8e6e6b6d0fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681397629952 2023-04-13 14:53:54,162 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,162 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,162 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 9db0b6253a5f45eb8068842c038dfdf5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681397630592 2023-04-13 14:53:54,163 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 5e0e0ce52a4f4434b3f25fea5f4e1ec0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1681397631229 2023-04-13 14:53:54,164 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:54,164 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:54,164 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 2e5ba16bbb914dbca1ce3118080abf77, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1681397632380 2023-04-13 14:53:54,165 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 96d18b6afdd84fa6a0dbdd6b43fd3473, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681397633518 2023-04-13 14:53:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,165 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,166 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:54,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:54,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:54,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:54,176 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:54,176 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:54,176 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:54,176 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:54,176 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:54,176 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:54,177 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:54,177 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-13 14:53:54,177 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:54,178 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:53:54,178 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:54,180 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:53:54,181 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:54,181 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:53:54,219 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#C#compaction#59 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:54,246 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/1b606f8c28fc40618950aeed61175b8a 2023-04-13 14:53:54,264 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/dcaf8a5f571d4e1bb1cd232ffc1e78e0 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/dcaf8a5f571d4e1bb1cd232ffc1e78e0 2023-04-13 14:53:54,277 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/C of 0d965d81bccf303fd24a24be9d6cbc05 into dcaf8a5f571d4e1bb1cd232ffc1e78e0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:54,277 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:54,277 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/C, priority=9, startTime=1681397634082; duration=0sec 2023-04-13 14:53:54,277 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:54,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:54,284 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:54,305 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/23d91be09c2348f3bf09557b19d6a74d 2023-04-13 14:53:54,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:54,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397694308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:54,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397694309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,336 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/3b1a4d9adbc44f11b77bb28bab150270 2023-04-13 14:53:54,344 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/1b606f8c28fc40618950aeed61175b8a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/1b606f8c28fc40618950aeed61175b8a 2023-04-13 14:53:54,351 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/1b606f8c28fc40618950aeed61175b8a, entries=150, sequenceid=381, filesize=11.9 K 2023-04-13 14:53:54,351 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/23d91be09c2348f3bf09557b19d6a74d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/23d91be09c2348f3bf09557b19d6a74d 2023-04-13 14:53:54,357 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/23d91be09c2348f3bf09557b19d6a74d, entries=150, sequenceid=381, filesize=11.9 K 2023-04-13 14:53:54,359 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/3b1a4d9adbc44f11b77bb28bab150270 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/3b1a4d9adbc44f11b77bb28bab150270 2023-04-13 14:53:54,368 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/3b1a4d9adbc44f11b77bb28bab150270, entries=150, sequenceid=381, filesize=11.9 K 2023-04-13 14:53:54,370 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 0d965d81bccf303fd24a24be9d6cbc05 in 193ms, sequenceid=381, compaction requested=false 2023-04-13 14:53:54,370 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:54,370 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:54,370 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:54,370 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:54,370 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,370 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:54,370 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:54,374 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,374 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:54,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75999f68[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:54,379 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:54,379 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,381 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,381 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:54,381 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:54,381 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:54,381 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,381 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:54,382 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:54,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:54,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:54,387 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:54,387 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:54,387 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:54,387 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:54,387 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:54,394 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,394 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:54,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,397 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:54,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:54,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:54,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,403 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:54,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:54,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:54,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,403 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:54,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:54,403 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:54,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:54,414 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 14:53:54,414 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:54,414 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:54,414 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:54,414 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:54,414 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:54,415 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:54,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:54,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397694448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:54,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397694451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,504 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:54,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:54,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:54,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:54,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:54,509 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:54,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:54,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:54,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:54,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,511 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:54,511 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:54,511 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,511 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:54,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:54,524 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:54,525 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:54,525 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:54,525 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:54,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:54,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:54,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:54,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:54,528 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,528 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:54,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ae977a7[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:54,538 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:54,538 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,539 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:54,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:54,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:54,541 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,541 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:54,547 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:54,547 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:54,547 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:54,547 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:54,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397694552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:54,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:54,555 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:54,555 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:54,555 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:54,555 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:54,555 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:54,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:54,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397694553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,557 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,557 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,557 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:54,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:54,557 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,559 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,563 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:54,563 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:54,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:54,563 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:54,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:54,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:54,564 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:54,564 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:54,564 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:54,563 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,564 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,564 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,564 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,564 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,564 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:54,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:54,664 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:54,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:54,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:54,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:54,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:54,672 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:54,672 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:54,672 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:54,672 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:54,673 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,673 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,675 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:54,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:54,675 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,675 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,675 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:54,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:54,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:54,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:54,680 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:54,681 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:54,681 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:54,681 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,682 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:54,682 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:54,683 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,683 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,684 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,684 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:54,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@465611d6[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:54,693 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:54,693 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,694 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,694 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:54,694 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:54,694 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,695 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:54,694 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:54,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:54,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:54,696 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:54,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:54,700 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:54,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:54,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:54,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:54,700 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:54,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,707 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,708 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,712 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:54,712 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:54,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:54,712 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:54,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:54,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:54,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:54,713 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:54,713 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:54,715 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,715 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:54,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:54,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397694756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:54,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397694758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,813 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:54,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:54,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:54,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:54,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:54,819 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:54,819 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:54,819 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:54,819 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:54,820 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,820 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,821 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:54,821 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:54,822 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,822 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:54,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:54,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:54,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:54,824 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:54,824 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:54,824 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:54,824 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:54,826 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,826 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,826 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,826 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:54,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2979c395[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:54,828 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:54,828 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,832 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/794866099cb2424ab58a5f37aff388b6 2023-04-13 14:53:54,834 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:54,835 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,835 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:54,836 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:54,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:54,843 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:54,843 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:54,843 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:54,843 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:54,843 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:54,844 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,844 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,844 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:54,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,846 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,848 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/08747af349c347f996621613b75f8455 2023-04-13 14:53:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,849 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:54,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:54,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:54,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:54,853 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:54,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:54,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:54,854 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,854 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,854 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,854 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,854 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,854 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:54,854 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:54,854 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:54,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:54,954 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:54,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:54,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:54,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:54,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:54,959 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:54,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:54,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:54,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:54,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,962 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:54,962 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:54,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:54,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,963 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,963 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:54,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:54,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:54,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:54,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:54,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:54,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:54,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:54,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:54,964 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:54,964 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:54,964 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:54,964 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:54,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:54,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:54,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:54,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:54,966 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,967 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,967 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:54,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10a26b0d[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:54,970 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:54,970 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,972 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:54,972 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:54,972 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:54,972 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:54,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:54,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:54,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:54,974 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,979 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:54,979 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:54,979 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:54,979 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:54,979 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:54,979 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:54,979 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:54,985 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,985 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:54,985 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:54,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:54,986 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,991 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:54,998 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:54,998 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:54,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:54,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:55,000 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,000 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:55,000 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:55,000 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,000 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,000 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,000 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:55,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:55,000 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:55,001 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:55,001 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:55,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:55,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:55,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397695059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:55,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397695061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,101 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:55,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:55,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:55,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:55,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:55,106 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:55,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:55,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:55,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:55,107 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,107 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:55,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,109 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:55,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:55,110 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,110 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:55,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:55,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:55,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:55,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:55,112 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:55,112 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:55,112 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:55,114 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:55,114 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:55,114 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:55,114 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:55,115 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:55,115 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:55,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:55,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:55,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:55,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:55,116 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,116 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:55,117 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,117 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:55,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:55,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@117772f1[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:55,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:55,120 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,124 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,124 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:55,126 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:55,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:55,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:55,135 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:55,135 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:55,135 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:55,135 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:55,135 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:55,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:55,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:55,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:55,138 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:55,141 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:55,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:55,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:55,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:55,150 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:55,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:55,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:55,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,150 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:55,150 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:55,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:55,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,151 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:55,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:55,250 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:55,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:55,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:55,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:55,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:55,255 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:55,255 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:55,255 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:55,255 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:55,255 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,255 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:55,257 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:55,257 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:55,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:55,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:55,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:55,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:55,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:55,260 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:55,260 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:55,260 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:55,260 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:55,261 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/93c1d9a99d7c4216851d0f39405572ab 2023-04-13 14:53:55,262 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,262 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:55,263 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,263 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:55,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:55,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@675919a8[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:55,266 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:55,266 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,267 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:55,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/794866099cb2424ab58a5f37aff388b6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/794866099cb2424ab58a5f37aff388b6 2023-04-13 14:53:55,268 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,269 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:55,269 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:55,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:55,272 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:55,272 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:55,272 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:55,272 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:55,272 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:55,273 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:55,273 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:55,273 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:55,276 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,276 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:55,276 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:55,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:55,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:55,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,278 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/794866099cb2424ab58a5f37aff388b6, entries=250, sequenceid=395, filesize=16.7 K 2023-04-13 14:53:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:55,278 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:55,279 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/08747af349c347f996621613b75f8455 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/08747af349c347f996621613b75f8455 2023-04-13 14:53:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:55,282 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,284 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:55,284 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:55,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:55,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:55,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:55,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:55,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:55,285 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:55,285 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:55,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:55,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:55,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:55,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:55,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:55,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:55,287 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/08747af349c347f996621613b75f8455, entries=150, sequenceid=395, filesize=11.9 K 2023-04-13 14:53:55,289 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/93c1d9a99d7c4216851d0f39405572ab as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/93c1d9a99d7c4216851d0f39405572ab 2023-04-13 14:53:55,306 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/93c1d9a99d7c4216851d0f39405572ab, entries=150, sequenceid=395, filesize=11.9 K 2023-04-13 14:53:55,308 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 0d965d81bccf303fd24a24be9d6cbc05 in 894ms, sequenceid=395, compaction requested=true 2023-04-13 14:53:55,308 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:55,308 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:55,309 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:53:55,309 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:53:55,310 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:55,310 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:55,311 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42086 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:53:55,311 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/A is initiating minor compaction (all files) 2023-04-13 14:53:55,311 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/A in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:55,311 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/397a23b4d83d45179c0905a64313c8cf, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/1b606f8c28fc40618950aeed61175b8a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/794866099cb2424ab58a5f37aff388b6] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=41.1 K 2023-04-13 14:53:55,311 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:53:55,312 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/B is initiating minor compaction (all files) 2023-04-13 14:53:55,312 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/B in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:55,312 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/1581635eab2e468eb8f5441064a1cb99, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/23d91be09c2348f3bf09557b19d6a74d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/08747af349c347f996621613b75f8455] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=36.3 K 2023-04-13 14:53:55,312 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 397a23b4d83d45179c0905a64313c8cf, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681397633518 2023-04-13 14:53:55,316 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 1b606f8c28fc40618950aeed61175b8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1681397633656 2023-04-13 14:53:55,317 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 1581635eab2e468eb8f5441064a1cb99, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681397633518 2023-04-13 14:53:55,317 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 794866099cb2424ab58a5f37aff388b6, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1681397634300 2023-04-13 14:53:55,317 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 23d91be09c2348f3bf09557b19d6a74d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1681397633656 2023-04-13 14:53:55,318 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 08747af349c347f996621613b75f8455, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1681397634305 2023-04-13 14:53:55,329 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#B#compaction#65 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:55,332 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#A#compaction#66 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:55,353 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/4d6a41b4e6dc48e1bed6dd06200133c3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/4d6a41b4e6dc48e1bed6dd06200133c3 2023-04-13 14:53:55,370 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/B of 0d965d81bccf303fd24a24be9d6cbc05 into 4d6a41b4e6dc48e1bed6dd06200133c3(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:55,371 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:55,371 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/B, priority=13, startTime=1681397635308; duration=0sec 2023-04-13 14:53:55,373 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:55,373 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:53:55,375 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:53:55,375 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/C is initiating minor compaction (all files) 2023-04-13 14:53:55,376 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/C in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:55,376 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/dcaf8a5f571d4e1bb1cd232ffc1e78e0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/3b1a4d9adbc44f11b77bb28bab150270, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/93c1d9a99d7c4216851d0f39405572ab] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=36.3 K 2023-04-13 14:53:55,377 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting dcaf8a5f571d4e1bb1cd232ffc1e78e0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681397633518 2023-04-13 14:53:55,378 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 3b1a4d9adbc44f11b77bb28bab150270, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1681397633656 2023-04-13 14:53:55,379 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 93c1d9a99d7c4216851d0f39405572ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1681397634305 2023-04-13 14:53:55,385 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:55,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:55,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:55,397 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#C#compaction#67 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:55,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:55,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:55,401 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:55,401 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:55,401 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:55,401 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:55,402 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,402 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:55,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:55,403 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:55,403 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,403 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:55,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:55,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:55,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:55,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:55,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:55,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:55,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:55,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:55,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:55,413 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:55,413 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:55,413 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 14:53:55,413 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:55,414 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:55,414 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:55,414 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:55,414 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:55,414 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:55,423 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/49fe1b2b6f7044248d71d7038f894705 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/49fe1b2b6f7044248d71d7038f894705 2023-04-13 14:53:55,437 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/039d9cb51e5b424a8ff2972abad0c3c5 2023-04-13 14:53:55,439 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/C of 0d965d81bccf303fd24a24be9d6cbc05 into 49fe1b2b6f7044248d71d7038f894705(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:55,439 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:55,439 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/C, priority=13, startTime=1681397635310; duration=0sec 2023-04-13 14:53:55,439 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:55,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:55,567 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:55,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:55,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397695580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:55,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397695582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:55,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397695683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:55,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397695687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,756 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/04fd11d693b14dc6bf01f690482fa2d1 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/04fd11d693b14dc6bf01f690482fa2d1 2023-04-13 14:53:55,763 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/A of 0d965d81bccf303fd24a24be9d6cbc05 into 04fd11d693b14dc6bf01f690482fa2d1(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:55,763 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:55,764 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/A, priority=13, startTime=1681397635308; duration=0sec 2023-04-13 14:53:55,764 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:55,873 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/9e8f173123464122a1ac28a18f3eb2b7 2023-04-13 14:53:55,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:55,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397695887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:55,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:55,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397695890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:56,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397696190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:56,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397696193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,286 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/03d2436b80f349be973f23be0c49fc17 2023-04-13 14:53:56,292 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/039d9cb51e5b424a8ff2972abad0c3c5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/039d9cb51e5b424a8ff2972abad0c3c5 2023-04-13 14:53:56,299 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/039d9cb51e5b424a8ff2972abad0c3c5, entries=150, sequenceid=415, filesize=11.9 K 2023-04-13 14:53:56,300 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/9e8f173123464122a1ac28a18f3eb2b7 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/9e8f173123464122a1ac28a18f3eb2b7 2023-04-13 14:53:56,305 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/9e8f173123464122a1ac28a18f3eb2b7, entries=150, sequenceid=415, filesize=11.9 K 2023-04-13 14:53:56,306 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/03d2436b80f349be973f23be0c49fc17 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/03d2436b80f349be973f23be0c49fc17 2023-04-13 14:53:56,311 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/03d2436b80f349be973f23be0c49fc17, entries=150, sequenceid=415, filesize=11.9 K 2023-04-13 14:53:56,313 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 0d965d81bccf303fd24a24be9d6cbc05 in 900ms, sequenceid=415, compaction requested=false 2023-04-13 14:53:56,313 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:56,313 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:56,313 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:56,313 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:56,313 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,313 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:56,313 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:56,315 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,315 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:56,315 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,315 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:56,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:56,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b1bbd[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:56,318 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:56,318 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,320 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,321 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:56,321 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:56,321 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,321 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:56,321 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:56,322 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,322 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:56,322 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:56,323 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:56,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:56,325 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:56,325 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:56,325 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:56,325 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:56,325 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:56,327 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,327 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,327 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:56,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,328 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,330 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,339 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,340 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:56,340 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:56,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:56,340 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:56,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:56,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:56,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:56,340 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:56,340 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:56,340 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:56,340 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,340 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:56,340 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,340 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,340 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:56,440 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:56,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:56,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:56,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:56,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:56,444 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:56,444 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:56,444 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:56,444 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:56,444 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,444 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:56,446 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,446 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:56,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:56,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:56,446 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,446 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:56,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:56,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:56,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:56,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:56,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:56,449 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:56,449 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:56,449 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:56,449 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:56,449 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:56,449 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:56,449 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:56,450 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 14:53:56,450 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:56,450 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:56,450 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:56,450 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:56,450 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:56,450 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:56,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:56,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:56,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:56,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397696721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:56,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397696722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:56,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397696828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:56,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397696830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,865 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/722639e18af749618e6da0567e226c60 2023-04-13 14:53:56,881 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/01ac152990dc49c99899f9cf88c8e9c0 2023-04-13 14:53:56,896 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/cbdae452714e4bdea1f6b225a0f82f0a 2023-04-13 14:53:56,907 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/722639e18af749618e6da0567e226c60 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/722639e18af749618e6da0567e226c60 2023-04-13 14:53:56,914 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/722639e18af749618e6da0567e226c60, entries=150, sequenceid=430, filesize=11.9 K 2023-04-13 14:53:56,915 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/01ac152990dc49c99899f9cf88c8e9c0 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/01ac152990dc49c99899f9cf88c8e9c0 2023-04-13 14:53:56,920 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/01ac152990dc49c99899f9cf88c8e9c0, entries=150, sequenceid=430, filesize=11.9 K 2023-04-13 14:53:56,921 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/cbdae452714e4bdea1f6b225a0f82f0a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/cbdae452714e4bdea1f6b225a0f82f0a 2023-04-13 14:53:56,951 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/cbdae452714e4bdea1f6b225a0f82f0a, entries=150, sequenceid=430, filesize=11.9 K 2023-04-13 14:53:56,952 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 0d965d81bccf303fd24a24be9d6cbc05 in 503ms, sequenceid=430, compaction requested=true 2023-04-13 14:53:56,952 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:56,952 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:56,952 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:56,952 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:56,952 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,952 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:56,952 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:56,954 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,954 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:56,955 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,955 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:56,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:56,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:56,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:56,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a2e2b8[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:56,958 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:56,958 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,959 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,959 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:56,959 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:56,959 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:56,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:56,962 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,962 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:56,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:56,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:56,962 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:56,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:56,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:56,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:56,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:56,969 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:56,969 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:56,969 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:56,969 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:56,969 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:56,971 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,971 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:56,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:56,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:56,971 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:56,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:56,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:56,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:56,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:56,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,978 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,979 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,982 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:56,983 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:56,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:56,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:56,983 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,983 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:56,983 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:56,983 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:56,983 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,983 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:56,983 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:56,983 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:56,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:56,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:56,984 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:56,984 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:56,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:57,033 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 14:53:57,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:57,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:57,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:57,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:57,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:57,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:57,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:57,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/6d49446aedeb4687b4608c40aa9faefd 2023-04-13 14:53:57,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:57,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397697062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:57,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397697063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,084 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:57,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:57,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:57,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:57,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:57,091 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:57,091 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:57,091 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:57,091 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:57,092 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,092 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,093 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,093 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:57,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:57,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:57,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,094 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,094 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:57,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:57,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:57,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:57,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:57,097 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:57,097 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:57,097 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:57,097 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:57,097 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:57,098 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:57,100 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:57,100 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:57,100 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:57,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:57,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:57,101 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,101 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:57,101 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:57,105 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,105 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,105 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,105 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:57,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2968f8c7[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:57,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:57,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,111 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,111 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:57,111 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:57,111 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:57,112 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,112 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:57,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:57,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:57,119 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:57,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:57,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:57,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:57,119 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:57,122 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,122 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,122 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:57,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,124 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,126 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,129 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:57,129 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:57,129 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,129 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:57,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:57,129 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:57,129 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:57,130 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:57,129 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,130 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,130 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,130 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:57,130 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:57,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:57,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:57,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397697165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:57,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397697167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,230 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:57,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:57,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:57,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:57,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:57,234 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:57,234 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:57,234 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:57,234 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:57,234 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,235 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,236 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:57,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:57,236 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,237 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,237 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:57,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:57,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:57,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:57,238 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:57,238 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:57,238 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:57,238 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:57,239 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:57,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,240 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:57,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@653301b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:57,242 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:57,242 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,243 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,244 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:57,244 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:57,244 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:57,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:57,246 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,246 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:57,246 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:57,246 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:57,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:57,253 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:57,253 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:57,253 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:57,253 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:57,253 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:57,255 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,255 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,255 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,257 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,260 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:57,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:57,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:57,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:57,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:57,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,264 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:57,265 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:57,265 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:57,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:57,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:57,365 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:57,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:57,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:57,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:57,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:57,369 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:57,369 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:57,369 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:57,369 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:57,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:57,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397697369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,374 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:57,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:57,375 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:57,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397697373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,376 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,376 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:57,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:57,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:57,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:57,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:57,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:57,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:57,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:57,377 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:57,377 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:57,377 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:57,377 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:57,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:57,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:57,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:57,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:57,379 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,379 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,379 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,379 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:57,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e0d559f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:57,381 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:57,381 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,382 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,382 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:57,382 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:57,382 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:57,382 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,382 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:57,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:57,384 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:57,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:57,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:57,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:57,389 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:57,389 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:57,389 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:57,389 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:57,389 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:57,392 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,392 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:57,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,392 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:57,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:57,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,393 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,394 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,398 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,398 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:57,398 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:57,398 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,398 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:57,398 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:57,398 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,398 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,398 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:57,398 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:57,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:57,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:57,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:57,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:57,399 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:57,399 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:57,483 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/74dfb4acdbbd4db19799e67dbbe3cbff 2023-04-13 14:53:57,498 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/e04e64346b664f30a69a2f6454ee72b9 2023-04-13 14:53:57,499 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:57,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:57,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:57,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:57,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:57,504 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:57,504 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:57,504 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:57,504 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:57,504 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/6d49446aedeb4687b4608c40aa9faefd as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6d49446aedeb4687b4608c40aa9faefd 2023-04-13 14:53:57,505 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,505 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,507 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:57,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:57,507 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:57,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:57,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:57,509 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:57,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6d49446aedeb4687b4608c40aa9faefd, entries=150, sequenceid=450, filesize=11.9 K 2023-04-13 14:53:57,509 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:57,509 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:57,509 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:57,510 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:57,510 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:57,510 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:57,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/74dfb4acdbbd4db19799e67dbbe3cbff as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/74dfb4acdbbd4db19799e67dbbe3cbff 2023-04-13 14:53:57,510 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:57,510 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:57,510 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:57,510 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:57,510 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,510 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:57,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,514 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,515 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/74dfb4acdbbd4db19799e67dbbe3cbff, entries=150, sequenceid=450, filesize=11.9 K 2023-04-13 14:53:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:57,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aaecbc4[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:57,516 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:57,516 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,518 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,518 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:57,518 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:57,518 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:57,518 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,518 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:57,519 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:57,519 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,519 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:57,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,520 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:57,520 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/e04e64346b664f30a69a2f6454ee72b9 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/e04e64346b664f30a69a2f6454ee72b9 2023-04-13 14:53:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:57,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:57,523 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:57,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:57,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:57,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:57,523 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:57,526 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/e04e64346b664f30a69a2f6454ee72b9, entries=150, sequenceid=450, filesize=11.9 K 2023-04-13 14:53:57,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:57,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,527 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 0d965d81bccf303fd24a24be9d6cbc05 in 495ms, sequenceid=450, compaction requested=true 2023-04-13 14:53:57,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,527 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:57,527 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:57,527 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:53:57,527 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,527 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:53:57,527 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:57,528 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,530 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:53:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,530 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/A is initiating minor compaction (all files) 2023-04-13 14:53:57,530 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:53:57,531 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/A in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:57,531 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/B is initiating minor compaction (all files) 2023-04-13 14:53:57,531 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/B in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:57,531 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/04fd11d693b14dc6bf01f690482fa2d1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/039d9cb51e5b424a8ff2972abad0c3c5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/722639e18af749618e6da0567e226c60, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6d49446aedeb4687b4608c40aa9faefd] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=48.3 K 2023-04-13 14:53:57,531 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/4d6a41b4e6dc48e1bed6dd06200133c3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/9e8f173123464122a1ac28a18f3eb2b7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/01ac152990dc49c99899f9cf88c8e9c0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/74dfb4acdbbd4db19799e67dbbe3cbff] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=48.3 K 2023-04-13 14:53:57,531 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 04fd11d693b14dc6bf01f690482fa2d1, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1681397634305 2023-04-13 14:53:57,531 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 4d6a41b4e6dc48e1bed6dd06200133c3, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1681397634305 2023-04-13 14:53:57,532 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 039d9cb51e5b424a8ff2972abad0c3c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681397634442 2023-04-13 14:53:57,532 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 9e8f173123464122a1ac28a18f3eb2b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681397634442 2023-04-13 14:53:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,532 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 722639e18af749618e6da0567e226c60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681397635573 2023-04-13 14:53:57,532 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,533 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 01ac152990dc49c99899f9cf88c8e9c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681397635573 2023-04-13 14:53:57,533 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 6d49446aedeb4687b4608c40aa9faefd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681397636719 2023-04-13 14:53:57,533 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 74dfb4acdbbd4db19799e67dbbe3cbff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681397636719 2023-04-13 14:53:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,536 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:57,536 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:57,536 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:57,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:57,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:57,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:57,536 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,537 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,537 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,537 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,537 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,537 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,537 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,537 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:57,537 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:57,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:57,549 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#B#compaction#77 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:57,558 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#A#compaction#78 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:57,572 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/83c324dc02684630b965b26824be9b74 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/83c324dc02684630b965b26824be9b74 2023-04-13 14:53:57,578 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/B of 0d965d81bccf303fd24a24be9d6cbc05 into 83c324dc02684630b965b26824be9b74(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:57,578 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:57,578 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/B, priority=12, startTime=1681397637527; duration=0sec 2023-04-13 14:53:57,579 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:57,579 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:53:57,581 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:53:57,581 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/C is initiating minor compaction (all files) 2023-04-13 14:53:57,581 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/C in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:57,581 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/49fe1b2b6f7044248d71d7038f894705, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/03d2436b80f349be973f23be0c49fc17, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/cbdae452714e4bdea1f6b225a0f82f0a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/e04e64346b664f30a69a2f6454ee72b9] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=48.3 K 2023-04-13 14:53:57,582 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 49fe1b2b6f7044248d71d7038f894705, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1681397634305 2023-04-13 14:53:57,582 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 03d2436b80f349be973f23be0c49fc17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681397634442 2023-04-13 14:53:57,582 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting cbdae452714e4bdea1f6b225a0f82f0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681397635573 2023-04-13 14:53:57,583 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting e04e64346b664f30a69a2f6454ee72b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681397636719 2023-04-13 14:53:57,594 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#C#compaction#79 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:57,611 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/af1b71c595c148128ef1cde8579637fc as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/af1b71c595c148128ef1cde8579637fc 2023-04-13 14:53:57,623 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/C of 0d965d81bccf303fd24a24be9d6cbc05 into af1b71c595c148128ef1cde8579637fc(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:57,623 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:57,623 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/C, priority=12, startTime=1681397637528; duration=0sec 2023-04-13 14:53:57,623 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:57,637 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:57,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:57,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:57,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:57,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:57,646 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:57,646 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:57,646 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:57,646 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:57,646 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,646 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,648 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,648 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:57,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:57,648 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,648 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:57,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:57,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:57,650 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:57,650 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:57,650 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:57,650 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:57,650 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:57,651 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:57,651 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:57,651 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 14:53:57,651 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:57,651 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:57,651 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:57,651 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:57,651 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:57,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:57,668 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/9cec3367cafc4c4383ea7dbdcaf40535 2023-04-13 14:53:57,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:57,679 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:57,687 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/67362e57a73a4bcd98f2bd156369d179 2023-04-13 14:53:57,724 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/489f7a0048424e24b4ad5ec820212475 2023-04-13 14:53:57,736 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/9cec3367cafc4c4383ea7dbdcaf40535 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/9cec3367cafc4c4383ea7dbdcaf40535 2023-04-13 14:53:57,746 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/9cec3367cafc4c4383ea7dbdcaf40535, entries=100, sequenceid=464, filesize=9.5 K 2023-04-13 14:53:57,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:57,747 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/67362e57a73a4bcd98f2bd156369d179 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/67362e57a73a4bcd98f2bd156369d179 2023-04-13 14:53:57,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397697743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:57,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397697746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,757 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/67362e57a73a4bcd98f2bd156369d179, entries=100, sequenceid=464, filesize=9.5 K 2023-04-13 14:53:57,758 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/489f7a0048424e24b4ad5ec820212475 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/489f7a0048424e24b4ad5ec820212475 2023-04-13 14:53:57,765 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/489f7a0048424e24b4ad5ec820212475, entries=100, sequenceid=464, filesize=9.5 K 2023-04-13 14:53:57,766 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 0d965d81bccf303fd24a24be9d6cbc05 in 115ms, sequenceid=464, compaction requested=false 2023-04-13 14:53:57,767 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:57,767 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:57,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:57,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:57,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:57,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:57,784 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,784 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,784 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,784 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:57,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@246eac96[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:57,786 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:57,786 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,788 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:57,788 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:57,788 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:57,788 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,788 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:57,792 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,793 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:57,793 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:57,793 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:57,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:57,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:57,796 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:57,796 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:57,796 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:57,796 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:57,796 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:57,798 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,798 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,798 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,799 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,801 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,804 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:57,804 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:57,804 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:57,804 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,804 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:57,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:57,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:57,804 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:57,804 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,805 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,805 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,805 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,805 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:57,805 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:57,852 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-13 14:53:57,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:57,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397637741, max timestamp=1681397637850Num uniques 859; 2023-04-13 14:53:57,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:57,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397637741, max timestamp=1681397637850Num uniques 859; 2023-04-13 14:53:57,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:57,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397637741, max timestamp=1681397637850Num uniques 859; 2023-04-13 14:53:57,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:57,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:57,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:53:57,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:57,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:53:57,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:57,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:53:57,885 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/0b672c71c1ff4e0dabbcb3524085001d 2023-04-13 14:53:57,905 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:57,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:57,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:57,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:57,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:57,908 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:57,908 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:57,908 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:57,909 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:57,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397637886, max timestamp=1681397637902Num uniques 859; 2023-04-13 14:53:57,909 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:57,909 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397637886, max timestamp=1681397637902Num uniques 859; 2023-04-13 14:53:57,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:57,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397637886, max timestamp=1681397637902Num uniques 859; 2023-04-13 14:53:57,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:53:57,910 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:57,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,911 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,911 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,911 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:57,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:57,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:57,913 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:57,913 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:57,913 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:57,914 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:57,914 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:57,914 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:57,914 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:57,916 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,916 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,917 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,917 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:57,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ba293e8[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:57,919 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:57,919 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,929 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,929 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,930 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:57,929 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:57,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:57,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:57,931 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,931 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,931 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:57,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:57,936 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:57,936 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:57,936 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:57,936 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:57,936 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:57,939 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,939 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,939 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:57,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:57,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:57,940 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,944 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,948 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:57,948 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:57,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:57,948 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:57,948 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:57,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:57,948 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,948 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:57,948 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:57,948 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,948 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:57,948 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:57,948 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:57,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:57,948 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:57,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681397637932, max timestamp=1681397637952Num uniques 1018; 2023-04-13 14:53:57,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-13 14:53:57,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681397637932, max timestamp=1681397637952Num uniques 1018; 2023-04-13 14:53:57,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-13 14:53:57,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681397637932, max timestamp=1681397637952Num uniques 1018; 2023-04-13 14:53:57,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-13 14:53:57,975 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/9c86f1354d714ef68b86a50b5e7b8f0e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/9c86f1354d714ef68b86a50b5e7b8f0e 2023-04-13 14:53:57,988 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/A of 0d965d81bccf303fd24a24be9d6cbc05 into 9c86f1354d714ef68b86a50b5e7b8f0e(size=12.8 K), total size for store is 22.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:57,988 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:57,988 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/A, priority=12, startTime=1681397637527; duration=0sec 2023-04-13 14:53:57,988 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:58,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681397637969, max timestamp=1681397638006Num uniques 1127; 2023-04-13 14:53:58,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-13 14:53:58,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681397637969, max timestamp=1681397638006Num uniques 1127; 2023-04-13 14:53:58,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681397637969, max timestamp=1681397638006Num uniques 1127; 2023-04-13 14:53:58,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-13 14:53:58,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-13 14:53:58,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681397638024, max timestamp=1681397638036Num uniques 1086; 2023-04-13 14:53:58,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-13 14:53:58,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681397638024, max timestamp=1681397638036Num uniques 1086; 2023-04-13 14:53:58,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681397638024, max timestamp=1681397638036Num uniques 1086; 2023-04-13 14:53:58,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-13 14:53:58,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-13 14:53:58,049 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:58,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:58,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:58,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:58,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:58,053 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:58,053 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:58,053 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:58,054 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:58,054 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,054 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,055 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:58,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:58,056 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,056 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,057 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:58,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:58,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:58,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:58,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:58,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:58,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:58,060 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:58,060 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:58,060 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:58,060 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:58,060 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:58,060 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:58,060 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:58,060 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:58,060 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,060 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:58,060 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:58,063 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,063 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:58,063 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,063 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:58,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:58,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:58,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:58,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f3798ae[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:58,065 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:58,065 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,068 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:58,068 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,068 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:58,070 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:58,070 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:58,070 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:58,072 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:58,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397698074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:58,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:58,078 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:58,078 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:58,078 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:58,078 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:58,078 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:58,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681397638056, max timestamp=1681397638075Num uniques 1145; 2023-04-13 14:53:58,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-13 14:53:58,080 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,080 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:58,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681397638056, max timestamp=1681397638075Num uniques 1145; 2023-04-13 14:53:58,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-13 14:53:58,082 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681397638056, max timestamp=1681397638075Num uniques 1145; 2023-04-13 14:53:58,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,080 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:58,084 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-13 14:53:58,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,088 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:58,089 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:58,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:58,089 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,089 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,089 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,089 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,089 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,089 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,089 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,089 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:58,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:58,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:58,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:58,090 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:58,090 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:58,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681397638130, max timestamp=1681397638152Num uniques 1054; 2023-04-13 14:53:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681397638130, max timestamp=1681397638152Num uniques 1054; 2023-04-13 14:53:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-13 14:53:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-13 14:53:58,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681397638130, max timestamp=1681397638152Num uniques 1054; 2023-04-13 14:53:58,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-13 14:53:58,190 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:58,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:58,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:58,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:58,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:58,197 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:58,197 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:58,197 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:58,197 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:58,197 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,197 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,200 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,200 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:58,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:58,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:58,200 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,200 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:58,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:58,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:58,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:58,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:58,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:58,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:58,219 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:58,219 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:58,219 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:58,219 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:58,219 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:58,219 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:58,219 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:58,219 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:58,219 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,219 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:58,219 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:58,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681397638200, max timestamp=1681397638217Num uniques 1013; 2023-04-13 14:53:58,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-04-13 14:53:58,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681397638200, max timestamp=1681397638217Num uniques 1013; 2023-04-13 14:53:58,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681397638200, max timestamp=1681397638217Num uniques 1013; 2023-04-13 14:53:58,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-04-13 14:53:58,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-04-13 14:53:58,221 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:58,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:58,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:58,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:58,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33fb8267[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:58,224 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:58,224 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,225 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,225 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:58,225 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:58,225 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:58,226 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,226 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:58,228 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:58,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:58,231 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:58,231 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:58,231 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:58,231 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:58,231 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:58,232 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,232 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:58,232 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:58,233 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,235 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,238 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:58,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:58,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:58,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:58,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,239 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:58,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:58,239 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:58,239 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:58,239 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:58,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:58,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681397638238, max timestamp=1681397638258Num uniques 1122; 2023-04-13 14:53:58,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681397638238, max timestamp=1681397638258Num uniques 1122; 2023-04-13 14:53:58,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-13 14:53:58,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-13 14:53:58,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681397638238, max timestamp=1681397638258Num uniques 1122; 2023-04-13 14:53:58,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-13 14:53:58,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1681397638278, max timestamp=1681397638290Num uniques 1081; 2023-04-13 14:53:58,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-13 14:53:58,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1681397638278, max timestamp=1681397638290Num uniques 1081; 2023-04-13 14:53:58,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1681397638278, max timestamp=1681397638290Num uniques 1081; 2023-04-13 14:53:58,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-13 14:53:58,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-13 14:53:58,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/fe30e185e28c4da8bfa5054dedb6e3f4 2023-04-13 14:53:58,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:58,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397698325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681397638313, max timestamp=1681397638325Num uniques 1190; 2023-04-13 14:53:58,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=10990, suffix heap size=151728,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=240 2023-04-13 14:53:58,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681397638313, max timestamp=1681397638325Num uniques 1190; 2023-04-13 14:53:58,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=10990, suffix heap size=151728,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=240 2023-04-13 14:53:58,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681397638313, max timestamp=1681397638325Num uniques 1190; 2023-04-13 14:53:58,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=10990, suffix heap size=151728,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=240 2023-04-13 14:53:58,339 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:58,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:58,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:58,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:58,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:58,342 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:58,342 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:58,342 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:58,342 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:58,343 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,343 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,344 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,344 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:58,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:58,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:58,344 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,344 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:58,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:58,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:58,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:58,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:58,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:58,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:58,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:58,346 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:58,346 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:58,346 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:58,346 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:58,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:58,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:58,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:58,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:58,349 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,349 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:58,350 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:58,350 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:58,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:58,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62a09292[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:58,351 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:58,351 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,352 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:58,353 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,353 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:58,357 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:58,357 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:58,357 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,357 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:58,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:58,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:58,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:58,359 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:58,359 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:58,359 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:58,359 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:58,359 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:58,363 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,363 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:58,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,363 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:58,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:58,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:58,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:58,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:58,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,365 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,366 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,369 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:58,369 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:58,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:58,369 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,369 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,369 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:58,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,372 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:58,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:58,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:58,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:58,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,373 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,373 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:58,373 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:58,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681397638360, max timestamp=1681397638391Num uniques 1099; 2023-04-13 14:53:58,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681397638360, max timestamp=1681397638391Num uniques 1099; 2023-04-13 14:53:58,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-13 14:53:58,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-13 14:53:58,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681397638360, max timestamp=1681397638391Num uniques 1099; 2023-04-13 14:53:58,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-13 14:53:58,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:58,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397698440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681397638435, max timestamp=1681397638440Num uniques 1108; 2023-04-13 14:53:58,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681397638429, max timestamp=1681397638440Num uniques 1108; 2023-04-13 14:53:58,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-13 14:53:58,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-13 14:53:58,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681397638435, max timestamp=1681397638440Num uniques 1108; 2023-04-13 14:53:58,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-13 14:53:58,473 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:58,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:58,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:58,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:58,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:58,480 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:58,480 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:58,480 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:58,480 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:58,480 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,481 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,482 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,482 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:58,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:58,482 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,482 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:58,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:58,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:58,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:58,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:58,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:58,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:58,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:58,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:58,484 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:58,484 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:58,484 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:58,484 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:58,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:58,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:58,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:58,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:58,487 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:58,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:58,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:58,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4db270fd[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:58,489 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:58,489 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,490 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,491 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:58,491 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:58,491 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:58,491 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,491 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:58,493 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:58,494 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:58,494 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:58,494 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:58,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681397638476, max timestamp=1681397638495Num uniques 1017; 2023-04-13 14:53:58,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-13 14:53:58,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681397638476, max timestamp=1681397638495Num uniques 1017; 2023-04-13 14:53:58,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-13 14:53:58,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:58,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681397638476, max timestamp=1681397638495Num uniques 1067; 2023-04-13 14:53:58,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-13 14:53:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:58,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:58,504 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:58,504 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:58,504 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:58,504 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:58,504 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:58,506 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,506 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:58,506 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,507 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:58,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:58,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:58,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,511 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,511 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,511 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,511 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,511 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,511 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,511 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,512 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:58,512 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:58,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:58,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:58,513 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:58,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:58,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:58,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:53:58,513 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:58,513 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:58,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681397638529, max timestamp=1681397638545Num uniques 1076; 2023-04-13 14:53:58,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681397638529, max timestamp=1681397638545Num uniques 1076; 2023-04-13 14:53:58,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-13 14:53:58,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-13 14:53:58,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681397638529, max timestamp=1681397638545Num uniques 1076; 2023-04-13 14:53:58,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-13 14:53:58,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1681397638564, max timestamp=1681397638577Num uniques 1035; 2023-04-13 14:53:58,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=15340, suffix heap size=135608,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=335 2023-04-13 14:53:58,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1681397638564, max timestamp=1681397638577Num uniques 1035; 2023-04-13 14:53:58,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=15340, suffix heap size=135608,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=335 2023-04-13 14:53:58,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1681397638564, max timestamp=1681397638577Num uniques 1035; 2023-04-13 14:53:58,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=15340, suffix heap size=135608,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=335 2023-04-13 14:53:58,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:58,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 446 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397698605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:58,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397698606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,614 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:58,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:58,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:58,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:58,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:58,618 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:58,618 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:58,618 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:58,618 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:58,618 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,619 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,620 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:58,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:58,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:58,620 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,620 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,620 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:58,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:58,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:58,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:58,622 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:58,622 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:58,622 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:58,622 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:58,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:58,623 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:58,623 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:58,623 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:58,623 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:58,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:58,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:58,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:58,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:58,625 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:58,626 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:58,626 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,626 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:58,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:58,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef3630d[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:58,629 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:58,629 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,630 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,630 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:58,630 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:58,630 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:58,630 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,630 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:58,631 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:58,631 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,631 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:58,632 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:58,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:58,634 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:58,634 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:58,634 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:58,634 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:58,634 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:58,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:58,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:58,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:58,637 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,638 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:58,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:58,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:58,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:58,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:58,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,643 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:58,643 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:58,643 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:58,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:58,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:58,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:58,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 448 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397698707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:58,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397698712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,718 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/9a7dd31fdff74a578cd7449614224d9d 2023-04-13 14:53:58,724 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/0b672c71c1ff4e0dabbcb3524085001d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0b672c71c1ff4e0dabbcb3524085001d 2023-04-13 14:53:58,731 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0b672c71c1ff4e0dabbcb3524085001d, entries=200, sequenceid=486, filesize=14.3 K 2023-04-13 14:53:58,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/fe30e185e28c4da8bfa5054dedb6e3f4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/fe30e185e28c4da8bfa5054dedb6e3f4 2023-04-13 14:53:58,738 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/fe30e185e28c4da8bfa5054dedb6e3f4, entries=150, sequenceid=486, filesize=11.9 K 2023-04-13 14:53:58,740 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/9a7dd31fdff74a578cd7449614224d9d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/9a7dd31fdff74a578cd7449614224d9d 2023-04-13 14:53:58,743 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:58,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:58,745 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/9a7dd31fdff74a578cd7449614224d9d, entries=150, sequenceid=486, filesize=11.9 K 2023-04-13 14:53:58,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:58,746 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=154.31 KB/158010 for 0d965d81bccf303fd24a24be9d6cbc05 in 894ms, sequenceid=486, compaction requested=true 2023-04-13 14:53:58,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:58,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:58,747 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:53:58,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:58,747 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:53:58,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:58,748 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37440 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:53:58,748 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/A is initiating minor compaction (all files) 2023-04-13 14:53:58,748 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35000 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:53:58,748 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/A in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:58,748 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/B is initiating minor compaction (all files) 2023-04-13 14:53:58,749 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/9c86f1354d714ef68b86a50b5e7b8f0e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/9cec3367cafc4c4383ea7dbdcaf40535, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0b672c71c1ff4e0dabbcb3524085001d] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=36.6 K 2023-04-13 14:53:58,749 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/B in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:58,749 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/83c324dc02684630b965b26824be9b74, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/67362e57a73a4bcd98f2bd156369d179, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/fe30e185e28c4da8bfa5054dedb6e3f4] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=34.2 K 2023-04-13 14:53:58,749 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 9c86f1354d714ef68b86a50b5e7b8f0e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681397636719 2023-04-13 14:53:58,749 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 83c324dc02684630b965b26824be9b74, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681397636719 2023-04-13 14:53:58,749 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 9cec3367cafc4c4383ea7dbdcaf40535, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1681397637054 2023-04-13 14:53:58,750 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 67362e57a73a4bcd98f2bd156369d179, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1681397637054 2023-04-13 14:53:58,750 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 0b672c71c1ff4e0dabbcb3524085001d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1681397637741 2023-04-13 14:53:58,750 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting fe30e185e28c4da8bfa5054dedb6e3f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1681397637741 2023-04-13 14:53:58,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:58,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:58,756 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:58,756 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:58,756 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:58,756 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:58,757 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,757 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,758 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:58,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:58,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:58,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:58,761 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#A#compaction#86 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:58,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:58,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:58,764 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:58,764 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:58,764 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:58,765 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#B#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:58,767 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:58,767 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:58,767 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=154.31 KB heapSize=428.25 KB 2023-04-13 14:53:58,768 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:58,768 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:53:58,768 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:58,768 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:53:58,768 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:58,768 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:53:58,784 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/89ca9602a7554df38b199c1e870156da as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/89ca9602a7554df38b199c1e870156da 2023-04-13 14:53:58,792 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/A of 0d965d81bccf303fd24a24be9d6cbc05 into 89ca9602a7554df38b199c1e870156da(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:58,792 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:58,792 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/A, priority=13, startTime=1681397638747; duration=0sec 2023-04-13 14:53:58,792 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:53:58,792 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:53:58,793 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:53:58,793 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/C is initiating minor compaction (all files) 2023-04-13 14:53:58,794 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/C in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:58,794 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/af1b71c595c148128ef1cde8579637fc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/489f7a0048424e24b4ad5ec820212475, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/9a7dd31fdff74a578cd7449614224d9d] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=34.1 K 2023-04-13 14:53:58,794 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting af1b71c595c148128ef1cde8579637fc, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681397636719 2023-04-13 14:53:58,795 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 489f7a0048424e24b4ad5ec820212475, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1681397637054 2023-04-13 14:53:58,796 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 9a7dd31fdff74a578cd7449614224d9d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1681397637741 2023-04-13 14:53:58,811 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=764 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/6df92750200c4ec7814c32f39321fd9a 2023-04-13 14:53:58,812 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#C#compaction#89 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:53:58,856 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/2d5a4ca251ce4fad82f550b4d08a180f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/2d5a4ca251ce4fad82f550b4d08a180f 2023-04-13 14:53:58,863 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/C of 0d965d81bccf303fd24a24be9d6cbc05 into 2d5a4ca251ce4fad82f550b4d08a180f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:58,863 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:58,863 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/C, priority=13, startTime=1681397638747; duration=0sec 2023-04-13 14:53:58,863 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:58,867 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=764 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/9f0f0d7f4874435781882053eee1a8e4 2023-04-13 14:53:58,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:58,921 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:58,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:58,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 423 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397698937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:58,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:58,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 453 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397698937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:59,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 425 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397699044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:59,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 455 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397699044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,199 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/d3da797792f54318866e6d621f508844 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/d3da797792f54318866e6d621f508844 2023-04-13 14:53:59,209 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/B of 0d965d81bccf303fd24a24be9d6cbc05 into d3da797792f54318866e6d621f508844(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:53:59,209 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:59,209 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/B, priority=13, startTime=1681397638747; duration=0sec 2023-04-13 14:53:59,209 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:53:59,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:59,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:59,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 427 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397699248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 457 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397699250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,333 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=764 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/836f60984ad94ffda3248f61f383c61e 2023-04-13 14:53:59,341 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/6df92750200c4ec7814c32f39321fd9a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6df92750200c4ec7814c32f39321fd9a 2023-04-13 14:53:59,346 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6df92750200c4ec7814c32f39321fd9a, entries=150, sequenceid=764, filesize=11.9 K 2023-04-13 14:53:59,348 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/9f0f0d7f4874435781882053eee1a8e4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/9f0f0d7f4874435781882053eee1a8e4 2023-04-13 14:53:59,353 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/9f0f0d7f4874435781882053eee1a8e4, entries=150, sequenceid=764, filesize=11.9 K 2023-04-13 14:53:59,354 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/836f60984ad94ffda3248f61f383c61e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/836f60984ad94ffda3248f61f383c61e 2023-04-13 14:53:59,361 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/836f60984ad94ffda3248f61f383c61e, entries=150, sequenceid=764, filesize=11.9 K 2023-04-13 14:53:59,362 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~428.20 KB/438480, currentSize=40.25 KB/41220 for 0d965d81bccf303fd24a24be9d6cbc05 in 595ms, sequenceid=764, compaction requested=false 2023-04-13 14:53:59,362 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:53:59,362 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:59,362 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:53:59,362 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:53:59,363 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,367 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:53:59,367 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:53:59,369 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:59,369 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:59,370 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:59,371 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:53:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:59,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:53:59,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11f42e0e[Count = 0] remaining members to acquire global barrier 2023-04-13 14:53:59,372 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:53:59,372 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:59,374 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:59,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:59,374 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:53:59,374 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:53:59,374 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:53:59,374 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,374 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:53:59,376 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:53:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:53:59,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:53:59,380 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:53:59,380 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:53:59,380 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:53:59,380 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:53:59,380 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:53:59,382 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:59,382 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:53:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:53:59,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:59,382 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:59,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:53:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:59,383 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:53:59,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:59,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,384 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:53:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:53:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:59,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:59,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:59,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:53:59,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:59,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:59,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:53:59,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:53:59,391 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:53:59,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:59,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:59,391 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:53:59,392 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:53:59,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:53:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:53:59,492 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:53:59,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:53:59,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:53:59,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:53:59,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:53:59,496 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:53:59,496 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:59,496 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:53:59,496 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:53:59,497 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:59,497 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:59,498 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,498 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:53:59,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:53:59,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:53:59,498 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,498 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:53:59,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:59,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:53:59,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:53:59,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:53:59,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:53:59,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:53:59,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:53:59,503 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:53:59,503 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:53:59,503 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:53:59,507 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:53:59,507 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:53:59,507 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:53:59,507 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:53:59,507 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 14:53:59,507 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:53:59,508 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:59,508 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:53:59,508 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:59,508 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:53:59,508 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:53:59,520 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/f3078e0c32b147d490d2eee871e3345d 2023-04-13 14:53:59,543 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/b93cc715ccea47259396d9bcab4a9bed 2023-04-13 14:53:59,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:53:59,560 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:53:59,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397639583, max timestamp=1681397639591Num uniques 859; 2023-04-13 14:53:59,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397639583, max timestamp=1681397639591Num uniques 859; 2023-04-13 14:53:59,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:53:59,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:53:59,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397639583, max timestamp=1681397639591Num uniques 859; 2023-04-13 14:53:59,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:53:59,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:59,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:59,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 447 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397699626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 475 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397699629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:59,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 477 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397699730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:59,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 449 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397699730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:59,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 479 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397699935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:53:59,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 451 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397699935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:53:59,961 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/a0609e3110244e4a9f2142867d7f9cdf 2023-04-13 14:53:59,971 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/f3078e0c32b147d490d2eee871e3345d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/f3078e0c32b147d490d2eee871e3345d 2023-04-13 14:53:59,982 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/f3078e0c32b147d490d2eee871e3345d, entries=150, sequenceid=776, filesize=11.9 K 2023-04-13 14:53:59,984 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/b93cc715ccea47259396d9bcab4a9bed as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/b93cc715ccea47259396d9bcab4a9bed 2023-04-13 14:53:59,992 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/b93cc715ccea47259396d9bcab4a9bed, entries=150, sequenceid=776, filesize=11.9 K 2023-04-13 14:53:59,996 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/a0609e3110244e4a9f2142867d7f9cdf as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/a0609e3110244e4a9f2142867d7f9cdf 2023-04-13 14:54:00,003 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/a0609e3110244e4a9f2142867d7f9cdf, entries=150, sequenceid=776, filesize=11.9 K 2023-04-13 14:54:00,004 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 0d965d81bccf303fd24a24be9d6cbc05 in 497ms, sequenceid=776, compaction requested=true 2023-04-13 14:54:00,004 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:54:00,004 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:54:00,004 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:00,004 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:00,004 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,004 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:00,004 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:00,006 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,006 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:00,006 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:00,006 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:00,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:00,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77a61aa0[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:00,008 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:00,008 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:00,009 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:00,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:00,010 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:00,010 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:00,010 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:00,010 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,010 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:00,012 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,013 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:00,013 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:00,013 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:00,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:00,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:00,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:00,017 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:00,017 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:00,017 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:00,017 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:00,017 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:00,020 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:00,020 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:00,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:00,020 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:00,021 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:00,023 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,026 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:00,026 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,026 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:00,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:00,026 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:00,026 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:00,026 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,026 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:00,026 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:00,027 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:00,026 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:00,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:00,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:00,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:00,027 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:00,027 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:00,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:00,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:00,127 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:00,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:00,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:00,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:00,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:00,131 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:00,131 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:00,131 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:00,131 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:00,131 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:00,132 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:00,134 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:00,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:00,134 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:00,134 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,134 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:00,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:00,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:00,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:00,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:00,135 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:00,135 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:00,135 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:00,136 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:00,136 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:00,136 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:54:00,136 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:54:00,136 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-13 14:54:00,136 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:54:00,136 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:00,137 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:54:00,137 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:00,137 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:54:00,137 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:00,151 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/88e9bb4576e14a2aaa819813eeeb2b88 2023-04-13 14:54:00,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:54:00,239 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:54:00,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:00,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397700246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:00,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 456 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397700246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:00,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 486 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397700349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:00,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 458 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397700349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:00,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 488 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397700551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:00,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 460 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397700553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,572 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/5e9d794f5bff46329b03f902e90d5009 2023-04-13 14:54:00,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:00,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44168 deadline: 1681397700742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,747 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18250 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:54:00,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:00,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44160 deadline: 1681397700758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,760 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18262 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:54:00,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:00,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44146 deadline: 1681397700758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,763 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18268 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:54:00,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:00,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 490 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397700858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:00,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 462 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397700859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:00,986 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/fe7ecc5f4daf4119ba2411e3fcfc0965 2023-04-13 14:54:00,995 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/88e9bb4576e14a2aaa819813eeeb2b88 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/88e9bb4576e14a2aaa819813eeeb2b88 2023-04-13 14:54:01,000 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/88e9bb4576e14a2aaa819813eeeb2b88, entries=150, sequenceid=813, filesize=11.9 K 2023-04-13 14:54:01,001 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/5e9d794f5bff46329b03f902e90d5009 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/5e9d794f5bff46329b03f902e90d5009 2023-04-13 14:54:01,006 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/5e9d794f5bff46329b03f902e90d5009, entries=150, sequenceid=813, filesize=11.9 K 2023-04-13 14:54:01,009 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/fe7ecc5f4daf4119ba2411e3fcfc0965 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/fe7ecc5f4daf4119ba2411e3fcfc0965 2023-04-13 14:54:01,014 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/fe7ecc5f4daf4119ba2411e3fcfc0965, entries=150, sequenceid=813, filesize=11.9 K 2023-04-13 14:54:01,015 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 0d965d81bccf303fd24a24be9d6cbc05 in 879ms, sequenceid=813, compaction requested=true 2023-04-13 14:54:01,015 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:54:01,015 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:54:01,015 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:01,015 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:01,015 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,015 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:01,015 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:01,017 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:01,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:01,018 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:01,018 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:01,018 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:01,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:01,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@294db931[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:01,019 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:01,019 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:01,020 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:01,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:01,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:01,020 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:01,021 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:01,021 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:01,021 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,021 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:01,022 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:01,022 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,022 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:01,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:01,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:01,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:01,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:01,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:01,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:01,025 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:01,025 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:01,025 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:01,025 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:01,025 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:01,026 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:01,026 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:01,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:01,026 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:01,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:01,027 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,029 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:01,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:01,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,031 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,031 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:01,031 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:01,031 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:01,031 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:01,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:01,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:01,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:01,031 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:01,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:01,031 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:01,031 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:01,031 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,032 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:01,032 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:01,032 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:01,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:01,132 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:01,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:01,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:01,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:01,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:01,145 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:01,145 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:01,145 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:01,145 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:01,145 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:01,145 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:01,146 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:01,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:01,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:01,147 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:01,147 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,147 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:01,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:01,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:01,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:01,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:01,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:01,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:01,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:01,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:01,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:01,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:01,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:01,148 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:54:01,148 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. started... 2023-04-13 14:54:01,149 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 14:54:01,149 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:54:01,149 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:01,149 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:54:01,149 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:01,149 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:54:01,149 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:01,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:54:01,362 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. as already flushing 2023-04-13 14:54:01,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397641386, max timestamp=1681397641392Num uniques 859; 2023-04-13 14:54:01,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397641386, max timestamp=1681397641392Num uniques 859; 2023-04-13 14:54:01,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397641386, max timestamp=1681397641392Num uniques 859; 2023-04-13 14:54:01,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:01,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:01,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:01,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:01,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:01,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 480 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397701422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 510 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397701422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:01,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 482 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397701526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:01,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 512 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397701526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,557 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=822 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/b05672c7de2640429377cfc999ad7d27 2023-04-13 14:54:01,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:01,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397701730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:01,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 514 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397701731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:01,970 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=822 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/dd96f251466448df97ac9e65bf3aa5c6 2023-04-13 14:54:01,975 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x60b8c4c7 to 127.0.0.1:62626 2023-04-13 14:54:01,975 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:01,975 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2879e2e1 to 127.0.0.1:62626 2023-04-13 14:54:01,975 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:01,979 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d24bbb7 to 127.0.0.1:62626 2023-04-13 14:54:01,979 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:01,980 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49d71120 to 127.0.0.1:62626 2023-04-13 14:54:01,980 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:02,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:02,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 486 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44164 deadline: 1681397702031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:02,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 516 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44182 deadline: 1681397702037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,385 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=822 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/7392bb0ab22b4df0a046286e0cfa4c81 2023-04-13 14:54:02,389 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/b05672c7de2640429377cfc999ad7d27 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/b05672c7de2640429377cfc999ad7d27 2023-04-13 14:54:02,393 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/b05672c7de2640429377cfc999ad7d27, entries=100, sequenceid=822, filesize=9.5 K 2023-04-13 14:54:02,393 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/dd96f251466448df97ac9e65bf3aa5c6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/dd96f251466448df97ac9e65bf3aa5c6 2023-04-13 14:54:02,397 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/dd96f251466448df97ac9e65bf3aa5c6, entries=100, sequenceid=822, filesize=9.5 K 2023-04-13 14:54:02,398 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/7392bb0ab22b4df0a046286e0cfa4c81 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/7392bb0ab22b4df0a046286e0cfa4c81 2023-04-13 14:54:02,402 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/7392bb0ab22b4df0a046286e0cfa4c81, entries=100, sequenceid=822, filesize=9.5 K 2023-04-13 14:54:02,403 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 0d965d81bccf303fd24a24be9d6cbc05 in 1255ms, sequenceid=822, compaction requested=true 2023-04-13 14:54:02,403 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:54:02,403 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:54:02,403 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:02,403 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:02,403 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,403 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:02,403 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:02,406 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:02,406 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:02,406 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:02,406 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:02,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:02,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cc34c1e[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:02,408 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:02,408 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:02,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:02,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:02,409 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:02,409 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:02,409 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:02,409 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,409 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:02,410 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:02,410 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,410 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:02,410 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:02,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:02,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:02,412 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:02,412 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:02,412 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:02,412 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:02,412 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:02,414 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:02,414 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:02,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:02,414 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:02,414 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:02,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:02,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:02,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:02,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:02,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,420 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,420 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:02,420 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:02,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:02,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:02,420 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:02,420 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:02,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:02,420 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:02,420 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:02,420 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:02,420 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:02,421 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:02,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:02,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:02,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:02,521 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:02,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:02,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:54:02,536 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-13 14:54:02,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:54:02,536 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e91d243 to 127.0.0.1:62626 2023-04-13 14:54:02,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:54:02,537 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:02,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:54:02,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:54:02,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:54:02,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:54:02,543 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30eb7fb2 to 127.0.0.1:62626 2023-04-13 14:54:02,543 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:02,945 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=860 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/68570dc84e194c939f2250269b4095b5 2023-04-13 14:54:03,374 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=860 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/552e366b766640688ba30895b70b39a0 2023-04-13 14:54:03,791 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=860 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/bf01bd494ec0404e9b28bebb6b0b021f 2023-04-13 14:54:03,796 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/68570dc84e194c939f2250269b4095b5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/68570dc84e194c939f2250269b4095b5 2023-04-13 14:54:03,800 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/68570dc84e194c939f2250269b4095b5, entries=150, sequenceid=860, filesize=11.9 K 2023-04-13 14:54:03,801 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/552e366b766640688ba30895b70b39a0 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/552e366b766640688ba30895b70b39a0 2023-04-13 14:54:03,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/552e366b766640688ba30895b70b39a0, entries=150, sequenceid=860, filesize=11.9 K 2023-04-13 14:54:03,806 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/bf01bd494ec0404e9b28bebb6b0b021f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/bf01bd494ec0404e9b28bebb6b0b021f 2023-04-13 14:54:03,810 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/bf01bd494ec0404e9b28bebb6b0b021f, entries=150, sequenceid=860, filesize=11.9 K 2023-04-13 14:54:03,811 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=6.71 KB/6870 for 0d965d81bccf303fd24a24be9d6cbc05 in 1274ms, sequenceid=860, compaction requested=true 2023-04-13 14:54:03,811 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:54:03,811 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:03,811 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:03,811 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 14:54:03,811 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 14:54:03,811 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:03,813 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71660 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 14:54:03,813 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71660 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 14:54:03,813 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/A is initiating minor compaction (all files) 2023-04-13 14:54:03,813 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/B is initiating minor compaction (all files) 2023-04-13 14:54:03,813 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/A in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:54:03,813 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/B in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:54:03,813 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/89ca9602a7554df38b199c1e870156da, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6df92750200c4ec7814c32f39321fd9a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/f3078e0c32b147d490d2eee871e3345d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/88e9bb4576e14a2aaa819813eeeb2b88, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/b05672c7de2640429377cfc999ad7d27, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/68570dc84e194c939f2250269b4095b5] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=70.0 K 2023-04-13 14:54:03,813 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/d3da797792f54318866e6d621f508844, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/9f0f0d7f4874435781882053eee1a8e4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/b93cc715ccea47259396d9bcab4a9bed, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/5e9d794f5bff46329b03f902e90d5009, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/dd96f251466448df97ac9e65bf3aa5c6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/552e366b766640688ba30895b70b39a0] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=70.0 K 2023-04-13 14:54:03,813 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 89ca9602a7554df38b199c1e870156da, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1681397637741 2023-04-13 14:54:03,814 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting d3da797792f54318866e6d621f508844, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1681397637741 2023-04-13 14:54:03,814 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 6df92750200c4ec7814c32f39321fd9a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=764, earliestPutTs=1681397638593 2023-04-13 14:54:03,814 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 9f0f0d7f4874435781882053eee1a8e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=764, earliestPutTs=1681397638593 2023-04-13 14:54:03,814 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting f3078e0c32b147d490d2eee871e3345d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=776, earliestPutTs=1681397638921 2023-04-13 14:54:03,814 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting b93cc715ccea47259396d9bcab4a9bed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=776, earliestPutTs=1681397638921 2023-04-13 14:54:03,814 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 88e9bb4576e14a2aaa819813eeeb2b88, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=813, earliestPutTs=1681397639608 2023-04-13 14:54:03,815 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 5e9d794f5bff46329b03f902e90d5009, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=813, earliestPutTs=1681397639608 2023-04-13 14:54:03,815 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting b05672c7de2640429377cfc999ad7d27, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=822, earliestPutTs=1681397640245 2023-04-13 14:54:03,815 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting dd96f251466448df97ac9e65bf3aa5c6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=822, earliestPutTs=1681397640245 2023-04-13 14:54:03,815 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 68570dc84e194c939f2250269b4095b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=860, earliestPutTs=1681397641416 2023-04-13 14:54:03,816 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 552e366b766640688ba30895b70b39a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=860, earliestPutTs=1681397641416 2023-04-13 14:54:03,828 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#B#compaction#104 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:03,829 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#A#compaction#105 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:03,837 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/b6aa73d09c4743caa5707d441ad1755c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/b6aa73d09c4743caa5707d441ad1755c 2023-04-13 14:54:03,842 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/A of 0d965d81bccf303fd24a24be9d6cbc05 into b6aa73d09c4743caa5707d441ad1755c(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:03,842 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:54:03,842 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/A, priority=10, startTime=1681397643811; duration=0sec 2023-04-13 14:54:03,842 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:03,842 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 14:54:03,844 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71626 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 14:54:03,844 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0d965d81bccf303fd24a24be9d6cbc05/C is initiating minor compaction (all files) 2023-04-13 14:54:03,845 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0d965d81bccf303fd24a24be9d6cbc05/C in TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:54:03,845 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/2d5a4ca251ce4fad82f550b4d08a180f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/836f60984ad94ffda3248f61f383c61e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/a0609e3110244e4a9f2142867d7f9cdf, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/fe7ecc5f4daf4119ba2411e3fcfc0965, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/7392bb0ab22b4df0a046286e0cfa4c81, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/bf01bd494ec0404e9b28bebb6b0b021f] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp, totalSize=69.9 K 2023-04-13 14:54:03,845 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 2d5a4ca251ce4fad82f550b4d08a180f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1681397637741 2023-04-13 14:54:03,848 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 836f60984ad94ffda3248f61f383c61e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=764, earliestPutTs=1681397638593 2023-04-13 14:54:03,848 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting a0609e3110244e4a9f2142867d7f9cdf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=776, earliestPutTs=1681397638921 2023-04-13 14:54:03,849 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting fe7ecc5f4daf4119ba2411e3fcfc0965, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=813, earliestPutTs=1681397639608 2023-04-13 14:54:03,849 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 7392bb0ab22b4df0a046286e0cfa4c81, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=822, earliestPutTs=1681397640245 2023-04-13 14:54:03,850 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting bf01bd494ec0404e9b28bebb6b0b021f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=860, earliestPutTs=1681397641416 2023-04-13 14:54:03,864 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0d965d81bccf303fd24a24be9d6cbc05#C#compaction#106 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:04,237 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/69171454343c4209aa5b0ac47598fefd as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/69171454343c4209aa5b0ac47598fefd 2023-04-13 14:54:04,242 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/B of 0d965d81bccf303fd24a24be9d6cbc05 into 69171454343c4209aa5b0ac47598fefd(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:04,242 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:54:04,242 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/B, priority=10, startTime=1681397643811; duration=0sec 2023-04-13 14:54:04,242 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:04,281 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/42f7655b5427458d9c0ebb6686689294 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/42f7655b5427458d9c0ebb6686689294 2023-04-13 14:54:04,285 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 0d965d81bccf303fd24a24be9d6cbc05/C of 0d965d81bccf303fd24a24be9d6cbc05 into 42f7655b5427458d9c0ebb6686689294(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:04,285 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:54:04,285 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05., storeName=0d965d81bccf303fd24a24be9d6cbc05/C, priority=10, startTime=1681397643811; duration=0sec 2023-04-13 14:54:04,286 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:10,790 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79eb61dd to 127.0.0.1:62626 2023-04-13 14:54:10,791 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:10,834 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x748f869b to 127.0.0.1:62626 2023-04-13 14:54:10,834 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:10,840 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46192d17 to 127.0.0.1:62626 2023-04-13 14:54:10,840 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:10,840 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 14:54:10,840 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 22 2023-04-13 14:54:10,840 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 13 2023-04-13 14:54:10,840 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 334 2023-04-13 14:54:10,841 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 366 2023-04-13 14:54:10,841 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 24 2023-04-13 14:54:10,841 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 14:54:10,841 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(398): read 4116 2023-04-13 14:54:10,841 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(398): read 4159 2023-04-13 14:54:10,841 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 14:54:10,841 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(402): scanned 1858 2023-04-13 14:54:10,841 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(403): verified 5565 rows 2023-04-13 14:54:10,841 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(402): scanned 1845 2023-04-13 14:54:10,841 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(403): verified 5533 rows 2023-04-13 14:54:10,841 INFO [Listener at localhost/37101] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 14:54:10,841 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d9fc901 to 127.0.0.1:62626 2023-04-13 14:54:10,841 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:10,845 INFO [Listener at localhost/37101] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 14:54:10,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 14:54:10,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:10,859 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397650859"}]},"ts":"1681397650859"} 2023-04-13 14:54:10,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 14:54:10,861 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 14:54:10,862 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 14:54:10,864 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0d965d81bccf303fd24a24be9d6cbc05, UNASSIGN}] 2023-04-13 14:54:10,865 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0d965d81bccf303fd24a24be9d6cbc05, UNASSIGN 2023-04-13 14:54:10,866 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=0d965d81bccf303fd24a24be9d6cbc05, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:10,866 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397650866"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397650866"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397650866"}]},"ts":"1681397650866"} 2023-04-13 14:54:10,867 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:54:10,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 14:54:11,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:54:11,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0d965d81bccf303fd24a24be9d6cbc05, disabling compactions & flushes 2023-04-13 14:54:11,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:54:11,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:54:11,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. after waiting 0 ms 2023-04-13 14:54:11,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:54:11,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 0d965d81bccf303fd24a24be9d6cbc05 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 14:54:11,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=A 2023-04-13 14:54:11,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:11,026 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=B 2023-04-13 14:54:11,026 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:11,026 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0d965d81bccf303fd24a24be9d6cbc05, store=C 2023-04-13 14:54:11,026 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:11,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 14:54:11,433 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=870 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/0b59fb2f4ab34ab381cc1c057dce6c01 2023-04-13 14:54:11,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 14:54:11,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=870 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/a938b80ae5fa4d528902f6b5e52abfe8 2023-04-13 14:54:11,880 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=870 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/728af32b506747d38e98e49b2e6d4e99 2023-04-13 14:54:11,886 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/A/0b59fb2f4ab34ab381cc1c057dce6c01 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0b59fb2f4ab34ab381cc1c057dce6c01 2023-04-13 14:54:11,891 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0b59fb2f4ab34ab381cc1c057dce6c01, entries=100, sequenceid=870, filesize=9.5 K 2023-04-13 14:54:11,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/B/a938b80ae5fa4d528902f6b5e52abfe8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/a938b80ae5fa4d528902f6b5e52abfe8 2023-04-13 14:54:11,905 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/a938b80ae5fa4d528902f6b5e52abfe8, entries=100, sequenceid=870, filesize=9.5 K 2023-04-13 14:54:11,906 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/.tmp/C/728af32b506747d38e98e49b2e6d4e99 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/728af32b506747d38e98e49b2e6d4e99 2023-04-13 14:54:11,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/728af32b506747d38e98e49b2e6d4e99, entries=100, sequenceid=870, filesize=9.5 K 2023-04-13 14:54:11,923 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 0d965d81bccf303fd24a24be9d6cbc05 in 898ms, sequenceid=870, compaction requested=false 2023-04-13 14:54:11,956 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/abd179e50af141b393d084431a796a20, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/62d5c7f0c3d64d2db3a52747c14757c9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/4ec279e6356f4eaa8756458ca0cda594, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/00c5158cf7934c21a1ebfa0cab5efa20, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/e4ab0937a1ea46e7a68475ac0f0e9193, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6defd5a7f3e24303baba8b05a414cc9c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/95d73b0890d249e985680135b441a8fc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/ce1aea1510ae4af7b00ecdfc83eb32f4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/2a98306c02c64fb88ef57dd3c99dc668, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0a004fcfcdd34fcf8f4837dadab9de01, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/03bf21dbbe8642ea8b667cd2a405d944, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/07136225dadd4b63b55a18395d1331a2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/70c6b49fead049f89996eb40d3886f8e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/b2bfd462b03f41d594f135cbbc56e8b1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/3466d1246f294bbb86d8a1a56ed321bb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/d2b4cde328b945caa371a85a61b92778, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/e932242b99a54841a07ed036a10dd0d5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/a6897d249815418c86e2e855c682d286, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/06bb15fd4bdd4c10b32f3733c331a695, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/397a23b4d83d45179c0905a64313c8cf, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/1b606f8c28fc40618950aeed61175b8a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/794866099cb2424ab58a5f37aff388b6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/04fd11d693b14dc6bf01f690482fa2d1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/039d9cb51e5b424a8ff2972abad0c3c5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/722639e18af749618e6da0567e226c60, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/9c86f1354d714ef68b86a50b5e7b8f0e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6d49446aedeb4687b4608c40aa9faefd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/9cec3367cafc4c4383ea7dbdcaf40535, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0b672c71c1ff4e0dabbcb3524085001d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/89ca9602a7554df38b199c1e870156da, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6df92750200c4ec7814c32f39321fd9a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/f3078e0c32b147d490d2eee871e3345d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/88e9bb4576e14a2aaa819813eeeb2b88, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/b05672c7de2640429377cfc999ad7d27, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/68570dc84e194c939f2250269b4095b5] to archive 2023-04-13 14:54:11,958 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:54:11,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 14:54:11,967 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/abd179e50af141b393d084431a796a20 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/abd179e50af141b393d084431a796a20 2023-04-13 14:54:11,969 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/62d5c7f0c3d64d2db3a52747c14757c9 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/62d5c7f0c3d64d2db3a52747c14757c9 2023-04-13 14:54:11,970 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/4ec279e6356f4eaa8756458ca0cda594 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/4ec279e6356f4eaa8756458ca0cda594 2023-04-13 14:54:11,972 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/00c5158cf7934c21a1ebfa0cab5efa20 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/00c5158cf7934c21a1ebfa0cab5efa20 2023-04-13 14:54:11,974 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/e4ab0937a1ea46e7a68475ac0f0e9193 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/e4ab0937a1ea46e7a68475ac0f0e9193 2023-04-13 14:54:11,976 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6defd5a7f3e24303baba8b05a414cc9c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6defd5a7f3e24303baba8b05a414cc9c 2023-04-13 14:54:11,978 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/95d73b0890d249e985680135b441a8fc to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/95d73b0890d249e985680135b441a8fc 2023-04-13 14:54:11,980 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/ce1aea1510ae4af7b00ecdfc83eb32f4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/ce1aea1510ae4af7b00ecdfc83eb32f4 2023-04-13 14:54:11,981 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/2a98306c02c64fb88ef57dd3c99dc668 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/2a98306c02c64fb88ef57dd3c99dc668 2023-04-13 14:54:11,983 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0a004fcfcdd34fcf8f4837dadab9de01 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0a004fcfcdd34fcf8f4837dadab9de01 2023-04-13 14:54:11,984 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/03bf21dbbe8642ea8b667cd2a405d944 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/03bf21dbbe8642ea8b667cd2a405d944 2023-04-13 14:54:11,986 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/07136225dadd4b63b55a18395d1331a2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/07136225dadd4b63b55a18395d1331a2 2023-04-13 14:54:11,987 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/70c6b49fead049f89996eb40d3886f8e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/70c6b49fead049f89996eb40d3886f8e 2023-04-13 14:54:11,989 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/b2bfd462b03f41d594f135cbbc56e8b1 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/b2bfd462b03f41d594f135cbbc56e8b1 2023-04-13 14:54:11,990 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/3466d1246f294bbb86d8a1a56ed321bb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/3466d1246f294bbb86d8a1a56ed321bb 2023-04-13 14:54:11,992 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/d2b4cde328b945caa371a85a61b92778 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/d2b4cde328b945caa371a85a61b92778 2023-04-13 14:54:11,993 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/e932242b99a54841a07ed036a10dd0d5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/e932242b99a54841a07ed036a10dd0d5 2023-04-13 14:54:11,994 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/a6897d249815418c86e2e855c682d286 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/a6897d249815418c86e2e855c682d286 2023-04-13 14:54:11,995 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/06bb15fd4bdd4c10b32f3733c331a695 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/06bb15fd4bdd4c10b32f3733c331a695 2023-04-13 14:54:11,996 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/397a23b4d83d45179c0905a64313c8cf to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/397a23b4d83d45179c0905a64313c8cf 2023-04-13 14:54:11,998 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/1b606f8c28fc40618950aeed61175b8a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/1b606f8c28fc40618950aeed61175b8a 2023-04-13 14:54:11,999 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/794866099cb2424ab58a5f37aff388b6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/794866099cb2424ab58a5f37aff388b6 2023-04-13 14:54:12,000 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/04fd11d693b14dc6bf01f690482fa2d1 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/04fd11d693b14dc6bf01f690482fa2d1 2023-04-13 14:54:12,002 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/039d9cb51e5b424a8ff2972abad0c3c5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/039d9cb51e5b424a8ff2972abad0c3c5 2023-04-13 14:54:12,003 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/722639e18af749618e6da0567e226c60 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/722639e18af749618e6da0567e226c60 2023-04-13 14:54:12,005 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/9c86f1354d714ef68b86a50b5e7b8f0e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/9c86f1354d714ef68b86a50b5e7b8f0e 2023-04-13 14:54:12,006 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6d49446aedeb4687b4608c40aa9faefd to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6d49446aedeb4687b4608c40aa9faefd 2023-04-13 14:54:12,007 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/9cec3367cafc4c4383ea7dbdcaf40535 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/9cec3367cafc4c4383ea7dbdcaf40535 2023-04-13 14:54:12,009 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0b672c71c1ff4e0dabbcb3524085001d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0b672c71c1ff4e0dabbcb3524085001d 2023-04-13 14:54:12,011 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/89ca9602a7554df38b199c1e870156da to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/89ca9602a7554df38b199c1e870156da 2023-04-13 14:54:12,012 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6df92750200c4ec7814c32f39321fd9a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/6df92750200c4ec7814c32f39321fd9a 2023-04-13 14:54:12,014 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/f3078e0c32b147d490d2eee871e3345d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/f3078e0c32b147d490d2eee871e3345d 2023-04-13 14:54:12,015 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/88e9bb4576e14a2aaa819813eeeb2b88 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/88e9bb4576e14a2aaa819813eeeb2b88 2023-04-13 14:54:12,016 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/b05672c7de2640429377cfc999ad7d27 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/b05672c7de2640429377cfc999ad7d27 2023-04-13 14:54:12,017 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/68570dc84e194c939f2250269b4095b5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/68570dc84e194c939f2250269b4095b5 2023-04-13 14:54:12,048 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/fc979feb74324bf7be8a081277fba9ff, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/07bace4873fc4c7089463d70a8d91af1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/50d0691c997a41f7a25f7b7df81245af, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/d8f753fabc364190bc2273e9dc5da922, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/eb308aa91da84aa79329c8c9627cbbdd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/3a284c2ac6184cfcac875af8ef0b793d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/847a268c61fb444793876829a81e6525, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/128fab1f7f7a4d4ea0af089f9246eb84, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/aec690d81f0941dcbfcea99d70d17a08, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/a09184d92f354fef9d1e278613e21ef3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/7ad98f9c12204d47863189d74e108668, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/a35837b696e34e17ab990269d34505fe, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/6541661e553c4cbb8adaf9e12389d231, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/e6422ef74ca341d99a6f245a104228ea, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/49170255b8c3485fb51c313bf125147e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/06411e287bde42428169cce4bf571e3f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/e302f001da414303b03257de81ebdcc2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/71bb9d05b2dd4730b913b5febb3c9a10, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/1581635eab2e468eb8f5441064a1cb99, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/3c74c93732544dbb813193e7f66640ec, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/23d91be09c2348f3bf09557b19d6a74d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/4d6a41b4e6dc48e1bed6dd06200133c3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/08747af349c347f996621613b75f8455, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/9e8f173123464122a1ac28a18f3eb2b7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/01ac152990dc49c99899f9cf88c8e9c0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/83c324dc02684630b965b26824be9b74, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/74dfb4acdbbd4db19799e67dbbe3cbff, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/67362e57a73a4bcd98f2bd156369d179, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/d3da797792f54318866e6d621f508844, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/fe30e185e28c4da8bfa5054dedb6e3f4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/9f0f0d7f4874435781882053eee1a8e4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/b93cc715ccea47259396d9bcab4a9bed, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/5e9d794f5bff46329b03f902e90d5009, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/dd96f251466448df97ac9e65bf3aa5c6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/552e366b766640688ba30895b70b39a0] to archive 2023-04-13 14:54:12,049 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:54:12,052 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/fc979feb74324bf7be8a081277fba9ff to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/fc979feb74324bf7be8a081277fba9ff 2023-04-13 14:54:12,053 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/07bace4873fc4c7089463d70a8d91af1 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/07bace4873fc4c7089463d70a8d91af1 2023-04-13 14:54:12,055 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/50d0691c997a41f7a25f7b7df81245af to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/50d0691c997a41f7a25f7b7df81245af 2023-04-13 14:54:12,056 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/d8f753fabc364190bc2273e9dc5da922 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/d8f753fabc364190bc2273e9dc5da922 2023-04-13 14:54:12,057 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/eb308aa91da84aa79329c8c9627cbbdd to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/eb308aa91da84aa79329c8c9627cbbdd 2023-04-13 14:54:12,059 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/3a284c2ac6184cfcac875af8ef0b793d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/3a284c2ac6184cfcac875af8ef0b793d 2023-04-13 14:54:12,060 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/847a268c61fb444793876829a81e6525 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/847a268c61fb444793876829a81e6525 2023-04-13 14:54:12,062 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/128fab1f7f7a4d4ea0af089f9246eb84 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/128fab1f7f7a4d4ea0af089f9246eb84 2023-04-13 14:54:12,063 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/aec690d81f0941dcbfcea99d70d17a08 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/aec690d81f0941dcbfcea99d70d17a08 2023-04-13 14:54:12,066 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/a09184d92f354fef9d1e278613e21ef3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/a09184d92f354fef9d1e278613e21ef3 2023-04-13 14:54:12,068 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/7ad98f9c12204d47863189d74e108668 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/7ad98f9c12204d47863189d74e108668 2023-04-13 14:54:12,069 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/a35837b696e34e17ab990269d34505fe to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/a35837b696e34e17ab990269d34505fe 2023-04-13 14:54:12,070 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/6541661e553c4cbb8adaf9e12389d231 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/6541661e553c4cbb8adaf9e12389d231 2023-04-13 14:54:12,072 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/e6422ef74ca341d99a6f245a104228ea to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/e6422ef74ca341d99a6f245a104228ea 2023-04-13 14:54:12,073 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/49170255b8c3485fb51c313bf125147e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/49170255b8c3485fb51c313bf125147e 2023-04-13 14:54:12,075 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/06411e287bde42428169cce4bf571e3f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/06411e287bde42428169cce4bf571e3f 2023-04-13 14:54:12,076 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/e302f001da414303b03257de81ebdcc2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/e302f001da414303b03257de81ebdcc2 2023-04-13 14:54:12,078 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/71bb9d05b2dd4730b913b5febb3c9a10 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/71bb9d05b2dd4730b913b5febb3c9a10 2023-04-13 14:54:12,079 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/1581635eab2e468eb8f5441064a1cb99 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/1581635eab2e468eb8f5441064a1cb99 2023-04-13 14:54:12,080 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/3c74c93732544dbb813193e7f66640ec to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/3c74c93732544dbb813193e7f66640ec 2023-04-13 14:54:12,082 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/23d91be09c2348f3bf09557b19d6a74d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/23d91be09c2348f3bf09557b19d6a74d 2023-04-13 14:54:12,083 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/4d6a41b4e6dc48e1bed6dd06200133c3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/4d6a41b4e6dc48e1bed6dd06200133c3 2023-04-13 14:54:12,085 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/08747af349c347f996621613b75f8455 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/08747af349c347f996621613b75f8455 2023-04-13 14:54:12,086 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/9e8f173123464122a1ac28a18f3eb2b7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/9e8f173123464122a1ac28a18f3eb2b7 2023-04-13 14:54:12,087 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/01ac152990dc49c99899f9cf88c8e9c0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/01ac152990dc49c99899f9cf88c8e9c0 2023-04-13 14:54:12,089 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/83c324dc02684630b965b26824be9b74 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/83c324dc02684630b965b26824be9b74 2023-04-13 14:54:12,090 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/74dfb4acdbbd4db19799e67dbbe3cbff to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/74dfb4acdbbd4db19799e67dbbe3cbff 2023-04-13 14:54:12,091 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/67362e57a73a4bcd98f2bd156369d179 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/67362e57a73a4bcd98f2bd156369d179 2023-04-13 14:54:12,093 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/d3da797792f54318866e6d621f508844 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/d3da797792f54318866e6d621f508844 2023-04-13 14:54:12,094 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/fe30e185e28c4da8bfa5054dedb6e3f4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/fe30e185e28c4da8bfa5054dedb6e3f4 2023-04-13 14:54:12,096 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/9f0f0d7f4874435781882053eee1a8e4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/9f0f0d7f4874435781882053eee1a8e4 2023-04-13 14:54:12,097 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/b93cc715ccea47259396d9bcab4a9bed to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/b93cc715ccea47259396d9bcab4a9bed 2023-04-13 14:54:12,099 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/5e9d794f5bff46329b03f902e90d5009 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/5e9d794f5bff46329b03f902e90d5009 2023-04-13 14:54:12,100 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/dd96f251466448df97ac9e65bf3aa5c6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/dd96f251466448df97ac9e65bf3aa5c6 2023-04-13 14:54:12,101 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/552e366b766640688ba30895b70b39a0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/552e366b766640688ba30895b70b39a0 2023-04-13 14:54:12,106 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/1f44c6dacf3e428190873301edc09497, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/e3fa0d10b5564bcebe6a5efe813a2d61, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/02e5f6ec55be4fbbb7bd29d7dc65f19f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/db91fd853c184bf5941540e94a50b945, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/bb9ba5c0e5d3498393d0095f3d2cf95a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/f8b96ee794644b6b849a9ffc3aff20e9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4b6348229394409b86553e719b244427, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4eaf886ea1784a509e6abcf67e9f802e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/74f4ab8a4ab34301a6b629aae9f1f1fa, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/ef80b0a5c3d54e12bd9d6e399ec38dae, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/f399207c8c7240ad8c1ced4d1f7ba298, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/08e460facade4839b31085498779b228, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/a4d1e13a40504fdfa896ae6b1dab4f60, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4886b370c79b4d76868dc8e6e6b6d0fe, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/9db0b6253a5f45eb8068842c038dfdf5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/5e0e0ce52a4f4434b3f25fea5f4e1ec0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/2e5ba16bbb914dbca1ce3118080abf77, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/dcaf8a5f571d4e1bb1cd232ffc1e78e0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/96d18b6afdd84fa6a0dbdd6b43fd3473, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/3b1a4d9adbc44f11b77bb28bab150270, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/49fe1b2b6f7044248d71d7038f894705, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/93c1d9a99d7c4216851d0f39405572ab, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/03d2436b80f349be973f23be0c49fc17, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/cbdae452714e4bdea1f6b225a0f82f0a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/af1b71c595c148128ef1cde8579637fc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/e04e64346b664f30a69a2f6454ee72b9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/489f7a0048424e24b4ad5ec820212475, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/2d5a4ca251ce4fad82f550b4d08a180f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/9a7dd31fdff74a578cd7449614224d9d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/836f60984ad94ffda3248f61f383c61e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/a0609e3110244e4a9f2142867d7f9cdf, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/fe7ecc5f4daf4119ba2411e3fcfc0965, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/7392bb0ab22b4df0a046286e0cfa4c81, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/bf01bd494ec0404e9b28bebb6b0b021f] to archive 2023-04-13 14:54:12,107 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:54:12,109 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/1f44c6dacf3e428190873301edc09497 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/1f44c6dacf3e428190873301edc09497 2023-04-13 14:54:12,111 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/e3fa0d10b5564bcebe6a5efe813a2d61 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/e3fa0d10b5564bcebe6a5efe813a2d61 2023-04-13 14:54:12,112 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/02e5f6ec55be4fbbb7bd29d7dc65f19f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/02e5f6ec55be4fbbb7bd29d7dc65f19f 2023-04-13 14:54:12,117 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/db91fd853c184bf5941540e94a50b945 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/db91fd853c184bf5941540e94a50b945 2023-04-13 14:54:12,119 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/bb9ba5c0e5d3498393d0095f3d2cf95a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/bb9ba5c0e5d3498393d0095f3d2cf95a 2023-04-13 14:54:12,120 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/f8b96ee794644b6b849a9ffc3aff20e9 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/f8b96ee794644b6b849a9ffc3aff20e9 2023-04-13 14:54:12,121 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4b6348229394409b86553e719b244427 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4b6348229394409b86553e719b244427 2023-04-13 14:54:12,122 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4eaf886ea1784a509e6abcf67e9f802e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4eaf886ea1784a509e6abcf67e9f802e 2023-04-13 14:54:12,123 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/74f4ab8a4ab34301a6b629aae9f1f1fa to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/74f4ab8a4ab34301a6b629aae9f1f1fa 2023-04-13 14:54:12,124 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/ef80b0a5c3d54e12bd9d6e399ec38dae to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/ef80b0a5c3d54e12bd9d6e399ec38dae 2023-04-13 14:54:12,126 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/f399207c8c7240ad8c1ced4d1f7ba298 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/f399207c8c7240ad8c1ced4d1f7ba298 2023-04-13 14:54:12,128 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/08e460facade4839b31085498779b228 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/08e460facade4839b31085498779b228 2023-04-13 14:54:12,129 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/a4d1e13a40504fdfa896ae6b1dab4f60 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/a4d1e13a40504fdfa896ae6b1dab4f60 2023-04-13 14:54:12,131 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4886b370c79b4d76868dc8e6e6b6d0fe to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/4886b370c79b4d76868dc8e6e6b6d0fe 2023-04-13 14:54:12,132 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/9db0b6253a5f45eb8068842c038dfdf5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/9db0b6253a5f45eb8068842c038dfdf5 2023-04-13 14:54:12,134 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/5e0e0ce52a4f4434b3f25fea5f4e1ec0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/5e0e0ce52a4f4434b3f25fea5f4e1ec0 2023-04-13 14:54:12,136 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/2e5ba16bbb914dbca1ce3118080abf77 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/2e5ba16bbb914dbca1ce3118080abf77 2023-04-13 14:54:12,138 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/dcaf8a5f571d4e1bb1cd232ffc1e78e0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/dcaf8a5f571d4e1bb1cd232ffc1e78e0 2023-04-13 14:54:12,139 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/96d18b6afdd84fa6a0dbdd6b43fd3473 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/96d18b6afdd84fa6a0dbdd6b43fd3473 2023-04-13 14:54:12,141 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/3b1a4d9adbc44f11b77bb28bab150270 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/3b1a4d9adbc44f11b77bb28bab150270 2023-04-13 14:54:12,142 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/49fe1b2b6f7044248d71d7038f894705 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/49fe1b2b6f7044248d71d7038f894705 2023-04-13 14:54:12,143 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/93c1d9a99d7c4216851d0f39405572ab to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/93c1d9a99d7c4216851d0f39405572ab 2023-04-13 14:54:12,145 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/03d2436b80f349be973f23be0c49fc17 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/03d2436b80f349be973f23be0c49fc17 2023-04-13 14:54:12,146 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/cbdae452714e4bdea1f6b225a0f82f0a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/cbdae452714e4bdea1f6b225a0f82f0a 2023-04-13 14:54:12,147 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/af1b71c595c148128ef1cde8579637fc to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/af1b71c595c148128ef1cde8579637fc 2023-04-13 14:54:12,148 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/e04e64346b664f30a69a2f6454ee72b9 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/e04e64346b664f30a69a2f6454ee72b9 2023-04-13 14:54:12,150 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/489f7a0048424e24b4ad5ec820212475 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/489f7a0048424e24b4ad5ec820212475 2023-04-13 14:54:12,151 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/2d5a4ca251ce4fad82f550b4d08a180f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/2d5a4ca251ce4fad82f550b4d08a180f 2023-04-13 14:54:12,153 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/9a7dd31fdff74a578cd7449614224d9d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/9a7dd31fdff74a578cd7449614224d9d 2023-04-13 14:54:12,154 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/836f60984ad94ffda3248f61f383c61e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/836f60984ad94ffda3248f61f383c61e 2023-04-13 14:54:12,156 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/a0609e3110244e4a9f2142867d7f9cdf to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/a0609e3110244e4a9f2142867d7f9cdf 2023-04-13 14:54:12,157 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/fe7ecc5f4daf4119ba2411e3fcfc0965 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/fe7ecc5f4daf4119ba2411e3fcfc0965 2023-04-13 14:54:12,159 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/7392bb0ab22b4df0a046286e0cfa4c81 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/7392bb0ab22b4df0a046286e0cfa4c81 2023-04-13 14:54:12,162 DEBUG [StoreCloser-TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/bf01bd494ec0404e9b28bebb6b0b021f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/bf01bd494ec0404e9b28bebb6b0b021f 2023-04-13 14:54:12,167 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/recovered.edits/873.seqid, newMaxSeqId=873, maxSeqId=1 2023-04-13 14:54:12,169 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05. 2023-04-13 14:54:12,169 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0d965d81bccf303fd24a24be9d6cbc05: 2023-04-13 14:54:12,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:54:12,172 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=0d965d81bccf303fd24a24be9d6cbc05, regionState=CLOSED 2023-04-13 14:54:12,172 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397652172"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397652172"}]},"ts":"1681397652172"} 2023-04-13 14:54:12,174 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-13 14:54:12,174 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 0d965d81bccf303fd24a24be9d6cbc05, server=jenkins-hbase4.apache.org,43875,1681397617638 in 1.3060 sec 2023-04-13 14:54:12,176 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-13 14:54:12,176 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0d965d81bccf303fd24a24be9d6cbc05, UNASSIGN in 1.3100 sec 2023-04-13 14:54:12,176 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397652176"}]},"ts":"1681397652176"} 2023-04-13 14:54:12,177 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 14:54:12,182 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 14:54:12,184 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.3310 sec 2023-04-13 14:54:12,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 14:54:12,965 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-13 14:54:12,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 14:54:12,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:12,979 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:12,981 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:12,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-13 14:54:12,986 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:54:12,990 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/recovered.edits] 2023-04-13 14:54:12,993 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0b59fb2f4ab34ab381cc1c057dce6c01 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/0b59fb2f4ab34ab381cc1c057dce6c01 2023-04-13 14:54:12,995 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/b6aa73d09c4743caa5707d441ad1755c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/A/b6aa73d09c4743caa5707d441ad1755c 2023-04-13 14:54:12,998 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/69171454343c4209aa5b0ac47598fefd to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/69171454343c4209aa5b0ac47598fefd 2023-04-13 14:54:12,999 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/a938b80ae5fa4d528902f6b5e52abfe8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/B/a938b80ae5fa4d528902f6b5e52abfe8 2023-04-13 14:54:13,001 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/42f7655b5427458d9c0ebb6686689294 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/42f7655b5427458d9c0ebb6686689294 2023-04-13 14:54:13,003 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/728af32b506747d38e98e49b2e6d4e99 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/C/728af32b506747d38e98e49b2e6d4e99 2023-04-13 14:54:13,005 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/recovered.edits/873.seqid to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05/recovered.edits/873.seqid 2023-04-13 14:54:13,006 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0d965d81bccf303fd24a24be9d6cbc05 2023-04-13 14:54:13,006 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 14:54:13,009 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:13,033 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 14:54:13,077 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 14:54:13,078 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:13,078 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 14:54:13,078 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681397653078"}]},"ts":"9223372036854775807"} 2023-04-13 14:54:13,080 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 14:54:13,081 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0d965d81bccf303fd24a24be9d6cbc05, NAME => 'TestAcidGuarantees,,1681397621182.0d965d81bccf303fd24a24be9d6cbc05.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 14:54:13,081 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 14:54:13,081 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681397653081"}]},"ts":"9223372036854775807"} 2023-04-13 14:54:13,083 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 14:54:13,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-13 14:54:13,085 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:13,086 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 114 msec 2023-04-13 14:54:13,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-13 14:54:13,284 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-13 14:54:13,301 INFO [Listener at localhost/37101] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=231 (was 208) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0xe3d782f-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:43875-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_752040327_15 at /127.0.0.1:50000 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0xe3d782f-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0xe3d782f-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0xe3d782f-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=536 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=361 (was 275) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=968 (was 1710) 2023-04-13 14:54:13,313 INFO [Listener at localhost/37101] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=231, OpenFileDescriptor=536, MaxFileDescriptor=60000, SystemLoadAverage=361, ProcessCount=169, AvailableMemoryMB=968 2023-04-13 14:54:13,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 14:54:13,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 14:54:13,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:13,319 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 14:54:13,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-13 14:54:13,320 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 14:54:13,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 14:54:13,324 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:13,325 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1 empty. 2023-04-13 14:54:13,325 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:13,325 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 14:54:13,335 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 14:54:13,337 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0b6ed6780bbc11018cb686e788422bc1, NAME => 'TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp 2023-04-13 14:54:13,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 14:54:13,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 14:54:13,744 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:54:13,744 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 0b6ed6780bbc11018cb686e788422bc1, disabling compactions & flushes 2023-04-13 14:54:13,744 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:13,745 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:13,745 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. after waiting 0 ms 2023-04-13 14:54:13,745 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:13,745 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:13,745 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:13,747 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 14:54:13,748 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397653748"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397653748"}]},"ts":"1681397653748"} 2023-04-13 14:54:13,749 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 14:54:13,750 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 14:54:13,750 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397653750"}]},"ts":"1681397653750"} 2023-04-13 14:54:13,751 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 14:54:13,754 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b6ed6780bbc11018cb686e788422bc1, ASSIGN}] 2023-04-13 14:54:13,756 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b6ed6780bbc11018cb686e788422bc1, ASSIGN 2023-04-13 14:54:13,757 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b6ed6780bbc11018cb686e788422bc1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43875,1681397617638; forceNewPlan=false, retain=false 2023-04-13 14:54:13,908 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=0b6ed6780bbc11018cb686e788422bc1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:13,908 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397653908"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397653908"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397653908"}]},"ts":"1681397653908"} 2023-04-13 14:54:13,910 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:54:13,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 14:54:14,065 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:14,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0b6ed6780bbc11018cb686e788422bc1, NAME => 'TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.', STARTKEY => '', ENDKEY => ''} 2023-04-13 14:54:14,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:14,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:54:14,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:14,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:14,067 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:14,069 DEBUG [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A 2023-04-13 14:54:14,069 DEBUG [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A 2023-04-13 14:54:14,069 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:54:14,069 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0b6ed6780bbc11018cb686e788422bc1 columnFamilyName A 2023-04-13 14:54:14,070 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.HStore(310): Store=0b6ed6780bbc11018cb686e788422bc1/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:54:14,070 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:14,071 DEBUG [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B 2023-04-13 14:54:14,071 DEBUG [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B 2023-04-13 14:54:14,071 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:54:14,072 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0b6ed6780bbc11018cb686e788422bc1 columnFamilyName B 2023-04-13 14:54:14,072 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.HStore(310): Store=0b6ed6780bbc11018cb686e788422bc1/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:54:14,072 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:14,074 DEBUG [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C 2023-04-13 14:54:14,074 DEBUG [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C 2023-04-13 14:54:14,074 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:54:14,074 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0b6ed6780bbc11018cb686e788422bc1 columnFamilyName C 2023-04-13 14:54:14,075 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.HStore(310): Store=0b6ed6780bbc11018cb686e788422bc1/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:54:14,075 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:14,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:14,076 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:14,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 14:54:14,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:14,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 14:54:14,081 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0b6ed6780bbc11018cb686e788422bc1; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60324490, jitterRate=-0.101095050573349}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 14:54:14,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:14,082 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., pid=18, masterSystemTime=1681397654061 2023-04-13 14:54:14,083 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:14,083 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:14,084 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=0b6ed6780bbc11018cb686e788422bc1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:14,084 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397654084"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681397654084"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681397654084"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681397654084"}]},"ts":"1681397654084"} 2023-04-13 14:54:14,087 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-13 14:54:14,087 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 in 175 msec 2023-04-13 14:54:14,089 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-13 14:54:14,089 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b6ed6780bbc11018cb686e788422bc1, ASSIGN in 333 msec 2023-04-13 14:54:14,089 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 14:54:14,089 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397654089"}]},"ts":"1681397654089"} 2023-04-13 14:54:14,090 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 14:54:14,094 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 14:54:14,095 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 778 msec 2023-04-13 14:54:14,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 14:54:14,425 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-13 14:54:14,426 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x0a607873 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:14,431 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31436cf6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:14,433 DEBUG [Listener at localhost/37101] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:14,437 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41098, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:14,438 DEBUG [Listener at localhost/37101] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 14:54:14,441 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 14:54:14,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 14:54:14,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 14:54:14,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:14,874 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 14:54:14,875 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 14:54:14,875 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 14:54:14,880 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-13 14:54:14,892 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b6ed6780bbc11018cb686e788422bc1, REOPEN/MOVE}] 2023-04-13 14:54:14,893 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b6ed6780bbc11018cb686e788422bc1, REOPEN/MOVE 2023-04-13 14:54:14,893 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=0b6ed6780bbc11018cb686e788422bc1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:14,893 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397654893"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397654893"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397654893"}]},"ts":"1681397654893"} 2023-04-13 14:54:14,895 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:54:15,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:15,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0b6ed6780bbc11018cb686e788422bc1, disabling compactions & flushes 2023-04-13 14:54:15,048 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:15,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:15,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. after waiting 0 ms 2023-04-13 14:54:15,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:15,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-13 14:54:15,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:15,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:15,052 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 0b6ed6780bbc11018cb686e788422bc1 to self. 2023-04-13 14:54:15,053 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:15,054 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=0b6ed6780bbc11018cb686e788422bc1, regionState=CLOSED 2023-04-13 14:54:15,054 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397655054"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397655054"}]},"ts":"1681397655054"} 2023-04-13 14:54:15,057 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-13 14:54:15,057 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 in 160 msec 2023-04-13 14:54:15,057 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b6ed6780bbc11018cb686e788422bc1, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,43875,1681397617638; forceNewPlan=false, retain=true 2023-04-13 14:54:15,208 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=0b6ed6780bbc11018cb686e788422bc1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:15,208 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397655208"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397655208"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397655208"}]},"ts":"1681397655208"} 2023-04-13 14:54:15,210 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:54:15,365 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:15,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0b6ed6780bbc11018cb686e788422bc1, NAME => 'TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.', STARTKEY => '', ENDKEY => ''} 2023-04-13 14:54:15,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:15,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:54:15,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:15,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:15,368 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:15,369 DEBUG [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A 2023-04-13 14:54:15,369 DEBUG [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A 2023-04-13 14:54:15,369 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:54:15,372 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0b6ed6780bbc11018cb686e788422bc1 columnFamilyName A 2023-04-13 14:54:15,374 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.HStore(310): Store=0b6ed6780bbc11018cb686e788422bc1/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:54:15,374 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:15,375 DEBUG [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B 2023-04-13 14:54:15,375 DEBUG [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B 2023-04-13 14:54:15,376 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:54:15,376 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0b6ed6780bbc11018cb686e788422bc1 columnFamilyName B 2023-04-13 14:54:15,377 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.HStore(310): Store=0b6ed6780bbc11018cb686e788422bc1/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:54:15,377 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:15,377 DEBUG [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C 2023-04-13 14:54:15,378 DEBUG [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C 2023-04-13 14:54:15,378 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:54:15,378 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0b6ed6780bbc11018cb686e788422bc1 columnFamilyName C 2023-04-13 14:54:15,379 INFO [StoreOpener-0b6ed6780bbc11018cb686e788422bc1-1] regionserver.HStore(310): Store=0b6ed6780bbc11018cb686e788422bc1/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:54:15,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:15,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:15,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:15,382 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 14:54:15,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:15,384 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0b6ed6780bbc11018cb686e788422bc1; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64141268, jitterRate=-0.04422062635421753}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 14:54:15,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:15,385 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., pid=23, masterSystemTime=1681397655361 2023-04-13 14:54:15,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:15,387 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:15,387 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=0b6ed6780bbc11018cb686e788422bc1, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:15,387 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397655387"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681397655387"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681397655387"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681397655387"}]},"ts":"1681397655387"} 2023-04-13 14:54:15,390 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-13 14:54:15,390 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 in 178 msec 2023-04-13 14:54:15,391 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-13 14:54:15,391 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b6ed6780bbc11018cb686e788422bc1, REOPEN/MOVE in 498 msec 2023-04-13 14:54:15,393 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-13 14:54:15,393 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 514 msec 2023-04-13 14:54:15,395 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 939 msec 2023-04-13 14:54:15,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-13 14:54:15,404 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x399105f2 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:15,408 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b3934fb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:15,409 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x55cbb115 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:15,413 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bbdba17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:15,413 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x0a71edf0 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:15,417 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2bab6b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:15,418 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x24bd3872 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:15,422 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73e51571, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:15,423 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x7aba8e46 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:15,426 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@370947cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:15,426 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x210d1008 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:15,430 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c472cfd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:15,430 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x00c2598b to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:15,435 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@498694cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:15,435 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x508246cc to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:15,438 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@421e5356, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:15,438 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x08795451 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:15,441 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a0fd7b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:15,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:15,456 DEBUG [hconnection-0x2dafd21e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:15,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:15,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:15,460 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:15,460 DEBUG [hconnection-0xd25198b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:15,460 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:15,461 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:15,461 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:15,461 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:15,461 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:15,462 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41100, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:15,462 DEBUG [hconnection-0x35c8190a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:15,462 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:15,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:15,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:15,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:15,463 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:15,463 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:15,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:15,464 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41104, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:15,464 DEBUG [hconnection-0x12abfe3b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:15,467 DEBUG [hconnection-0x33449c69-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:15,467 DEBUG [hconnection-0x27d25b6c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:15,468 DEBUG [hconnection-0x479f41c8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:15,472 DEBUG [hconnection-0x3ef18615-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:15,472 DEBUG [hconnection-0x62f3bd1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:15,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:15,477 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41106, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:15,477 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:15,478 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41128, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:15,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:15,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:15,480 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41142, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:15,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:15,481 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41140, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:15,482 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41154, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:15,483 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:15,483 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41148, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:15,483 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:15,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:15,485 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:15,488 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:15,488 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:15,488 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:15,493 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-13 14:54:15,497 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:15,497 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:15,497 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:15,497 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:15,497 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:15,497 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:15,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:15,503 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:15,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397655507, max timestamp=1681397655523Num uniques 859; 2023-04-13 14:54:15,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:15,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397655507, max timestamp=1681397655523Num uniques 859; 2023-04-13 14:54:15,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:15,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397655507, max timestamp=1681397655523Num uniques 859; 2023-04-13 14:54:15,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:15,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397655589, max timestamp=1681397655594Num uniques 1068; 2023-04-13 14:54:15,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-13 14:54:15,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397655589, max timestamp=1681397655594Num uniques 1068; 2023-04-13 14:54:15,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-13 14:54:15,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397655589, max timestamp=1681397655594Num uniques 1068; 2023-04-13 14:54:15,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=6870, suffix heap size=139040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=150 2023-04-13 14:54:15,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681397655606, max timestamp=1681397655614Num uniques 1027; 2023-04-13 14:54:15,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 14:54:15,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681397655606, max timestamp=1681397655614Num uniques 1027; 2023-04-13 14:54:15,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 14:54:15,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681397655606, max timestamp=1681397655614Num uniques 1009; 2023-04-13 14:54:15,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-04-13 14:54:15,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681397655626, max timestamp=1681397655638Num uniques 1036; 2023-04-13 14:54:15,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-04-13 14:54:15,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681397655626, max timestamp=1681397655638Num uniques 1036; 2023-04-13 14:54:15,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-04-13 14:54:15,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681397655631, max timestamp=1681397655638Num uniques 1036; 2023-04-13 14:54:15,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-13 14:54:15,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681397655650, max timestamp=1681397655664Num uniques 1195; 2023-04-13 14:54:15,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=15800, suffix heap size=152248,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=345 2023-04-13 14:54:15,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681397655650, max timestamp=1681397655664Num uniques 1045; 2023-04-13 14:54:15,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=15800, suffix heap size=136648,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=345 2023-04-13 14:54:15,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681397655650, max timestamp=1681397655664Num uniques 1195; 2023-04-13 14:54:15,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=15800, suffix heap size=152248,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=345 2023-04-13 14:54:15,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681397655677, max timestamp=1681397655712Num uniques 1204; 2023-04-13 14:54:15,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681397655677, max timestamp=1681397655712Num uniques 1204; 2023-04-13 14:54:15,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-04-13 14:54:15,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681397655677, max timestamp=1681397655712Num uniques 1204; 2023-04-13 14:54:15,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-04-13 14:54:15,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-04-13 14:54:15,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:15,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41140 deadline: 1681397715733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:15,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:15,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41104 deadline: 1681397715733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:15,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:15,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1681397655726, max timestamp=1681397655733Num uniques 1113; 2023-04-13 14:54:15,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=14328, suffix heap size=143720,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=313 2023-04-13 14:54:15,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681397655731, max timestamp=1681397655733Num uniques 1113; 2023-04-13 14:54:15,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-13 14:54:15,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41142 deadline: 1681397715734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:15,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681397655731, max timestamp=1681397655733Num uniques 1113; 2023-04-13 14:54:15,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-13 14:54:15,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681397655755, max timestamp=1681397655763Num uniques 1022; 2023-04-13 14:54:15,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-13 14:54:15,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681397655755, max timestamp=1681397655763Num uniques 1172; 2023-04-13 14:54:15,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=12452, suffix heap size=149856,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=272 2023-04-13 14:54:15,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681397655755, max timestamp=1681397655763Num uniques 1022; 2023-04-13 14:54:15,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-13 14:54:15,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=381, cellSize=17446, totalHeapSize=39864, min timestamp=1681397655783, max timestamp=1681397655794Num uniques 1131; 2023-04-13 14:54:15,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=381, cellSize=17446, totalHeapSize=39864, min timestamp=1681397655783, max timestamp=1681397655794Num uniques 1131; 2023-04-13 14:54:15,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=17446, suffix heap size=145592,new segment heap size=39864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=381 2023-04-13 14:54:15,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=381, cellSize=17446, totalHeapSize=39864, min timestamp=1681397655783, max timestamp=1681397655794Num uniques 1131; 2023-04-13 14:54:15,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=17446, suffix heap size=145592,new segment heap size=39864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=381 2023-04-13 14:54:15,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=17446, suffix heap size=145592,new segment heap size=39864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=381 2023-04-13 14:54:15,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681397655820, max timestamp=1681397655832Num uniques 1240; 2023-04-13 14:54:15,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681397655820, max timestamp=1681397655832Num uniques 1240; 2023-04-13 14:54:15,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681397655820, max timestamp=1681397655832Num uniques 1240; 2023-04-13 14:54:15,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=56790, new segment data size=13280, suffix heap size=156928,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1240, new segment cells count=290 2023-04-13 14:54:15,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=56790, new segment data size=13280, suffix heap size=156928,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1240, new segment cells count=290 2023-04-13 14:54:15,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=56790, new segment data size=13280, suffix heap size=156928,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1240, new segment cells count=290 2023-04-13 14:54:15,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681397655845, max timestamp=1681397655854Num uniques 1149; 2023-04-13 14:54:15,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-13 14:54:15,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681397655845, max timestamp=1681397655854Num uniques 1149; 2023-04-13 14:54:15,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-13 14:54:15,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681397655845, max timestamp=1681397655854Num uniques 1149; 2023-04-13 14:54:15,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-13 14:54:15,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681397655869, max timestamp=1681397655879Num uniques 1108; 2023-04-13 14:54:15,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681397655869, max timestamp=1681397655879Num uniques 1108; 2023-04-13 14:54:15,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-13 14:54:15,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681397655869, max timestamp=1681397655879Num uniques 1108; 2023-04-13 14:54:15,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-13 14:54:15,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-13 14:54:15,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=367, cellSize=16802, totalHeapSize=38408, min timestamp=1681397655891, max timestamp=1681397655901Num uniques 1017; 2023-04-13 14:54:15,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=16802, suffix heap size=133736,new segment heap size=38408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=367 2023-04-13 14:54:15,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397655892, max timestamp=1681397655901Num uniques 1017; 2023-04-13 14:54:15,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=6870, suffix heap size=133736,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=150 2023-04-13 14:54:15,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397655892, max timestamp=1681397655901Num uniques 1017; 2023-04-13 14:54:15,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=6870, suffix heap size=133736,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=150 2023-04-13 14:54:15,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=376, cellSize=17216, totalHeapSize=39344, min timestamp=1681397655914, max timestamp=1681397655924Num uniques 1226; 2023-04-13 14:54:15,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=376, cellSize=17216, totalHeapSize=39344, min timestamp=1681397655914, max timestamp=1681397655924Num uniques 1009; 2023-04-13 14:54:15,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=56146, new segment data size=17216, suffix heap size=155472,new segment heap size=39344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1226, new segment cells count=376 2023-04-13 14:54:15,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=376, cellSize=17216, totalHeapSize=39344, min timestamp=1681397655914, max timestamp=1681397655924Num uniques 1009; 2023-04-13 14:54:15,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=17216, suffix heap size=132904,new segment heap size=39344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=376 2023-04-13 14:54:15,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=17216, suffix heap size=132904,new segment heap size=39344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=376 2023-04-13 14:54:15,958 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413e2181ca0b4b64dbe874af3332e217ceb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e2181ca0b4b64dbe874af3332e217ceb 2023-04-13 14:54:15,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10750, totalHeapSize=24680, min timestamp=1681397655939, max timestamp=1681397655951Num uniques 1235; 2023-04-13 14:54:15,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=56550, new segment data size=10750, suffix heap size=156408,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1235, new segment cells count=235 2023-04-13 14:54:15,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10750, totalHeapSize=24680, min timestamp=1681397655939, max timestamp=1681397655951Num uniques 1235; 2023-04-13 14:54:15,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=56550, new segment data size=10750, suffix heap size=156408,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1235, new segment cells count=235 2023-04-13 14:54:15,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=191, cellSize=8746, totalHeapSize=20104, min timestamp=1681397655939, max timestamp=1681397655951Num uniques 1235; 2023-04-13 14:54:15,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=56560, new segment data size=8746, suffix heap size=156408,new segment heap size=20104 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1235, new segment cells count=191 2023-04-13 14:54:15,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=494, cellSize=22624, totalHeapSize=51616, min timestamp=1681397655973, max timestamp=1681397655992Num uniques 1094; 2023-04-13 14:54:15,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=22624, suffix heap size=141744,new segment heap size=51616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=494 2023-04-13 14:54:15,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=494, cellSize=22624, totalHeapSize=51616, min timestamp=1681397655973, max timestamp=1681397655992Num uniques 1094; 2023-04-13 14:54:15,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=22624, suffix heap size=141744,new segment heap size=51616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=494 2023-04-13 14:54:15,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681397655973, max timestamp=1681397655992Num uniques 1050; 2023-04-13 14:54:15,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48090, new segment data size=11174, suffix heap size=137168,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1050, new segment cells count=244 2023-04-13 14:54:16,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41122 deadline: 1681397716045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41104 deadline: 1681397716045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1681397656038, max timestamp=1681397656045Num uniques 1353; 2023-04-13 14:54:16,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9293, totalHeapSize=21352, min timestamp=1681397656038, max timestamp=1681397656045Num uniques 1353; 2023-04-13 14:54:16,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=61963, new segment data size=9293, suffix heap size=168680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1353, new segment cells count=203 2023-04-13 14:54:16,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397656038, max timestamp=1681397656046Num uniques 1103; 2023-04-13 14:54:16,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50500, new segment data size=6870, suffix heap size=142680,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=150 2023-04-13 14:54:16,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=61958, new segment data size=9288, suffix heap size=168680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1353, new segment cells count=203 2023-04-13 14:54:16,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681397656057, max timestamp=1681397656070Num uniques 1062; 2023-04-13 14:54:16,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=11992, suffix heap size=138416,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=262 2023-04-13 14:54:16,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681397656057, max timestamp=1681397656070Num uniques 1062; 2023-04-13 14:54:16,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=11992, suffix heap size=138416,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=262 2023-04-13 14:54:16,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681397656057, max timestamp=1681397656070Num uniques 1009; 2023-04-13 14:54:16,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46222, new segment data size=11992, suffix heap size=132904,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=262 2023-04-13 14:54:16,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681397656082, max timestamp=1681397656120Num uniques 1121; 2023-04-13 14:54:16,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=12406, suffix heap size=144552,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=271 2023-04-13 14:54:16,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681397656082, max timestamp=1681397656120Num uniques 1121; 2023-04-13 14:54:16,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=12406, suffix heap size=144552,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=271 2023-04-13 14:54:16,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681397656082, max timestamp=1681397656120Num uniques 1121; 2023-04-13 14:54:16,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=12406, suffix heap size=144552,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=271 2023-04-13 14:54:16,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1681397656129, max timestamp=1681397656137Num uniques 1130; 2023-04-13 14:54:16,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=10530, suffix heap size=145488,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=230 2023-04-13 14:54:16,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681397656129, max timestamp=1681397656137Num uniques 1130; 2023-04-13 14:54:16,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=12820, suffix heap size=145488,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=280 2023-04-13 14:54:16,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1681397656129, max timestamp=1681397656137Num uniques 1130; 2023-04-13 14:54:16,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=10530, suffix heap size=145488,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=230 2023-04-13 14:54:16,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1681397656153, max timestamp=1681397656174Num uniques 1089; 2023-04-13 14:54:16,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=13234, suffix heap size=141224,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=289 2023-04-13 14:54:16,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1681397656153, max timestamp=1681397656174Num uniques 1139; 2023-04-13 14:54:16,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=13234, suffix heap size=146424,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=289 2023-04-13 14:54:16,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1681397656153, max timestamp=1681397656174Num uniques 1089; 2023-04-13 14:54:16,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=13234, suffix heap size=141224,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=289 2023-04-13 14:54:16,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1681397656185, max timestamp=1681397656189Num uniques 1148; 2023-04-13 14:54:16,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=13648, suffix heap size=147360,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=298 2023-04-13 14:54:16,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1681397656185, max timestamp=1681397656189Num uniques 1148; 2023-04-13 14:54:16,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=13648, suffix heap size=147360,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=298 2023-04-13 14:54:16,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1681397656185, max timestamp=1681397656189Num uniques 1148; 2023-04-13 14:54:16,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=13648, suffix heap size=147360,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=298 2023-04-13 14:54:16,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41140 deadline: 1681397716201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1681397656199, max timestamp=1681397656201Num uniques 1157; 2023-04-13 14:54:16,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1681397656199, max timestamp=1681397656201Num uniques 1157; 2023-04-13 14:54:16,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=9475, suffix heap size=148296,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=207 2023-04-13 14:54:16,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=9475, suffix heap size=148296,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=207 2023-04-13 14:54:16,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1681397656199, max timestamp=1681397656201Num uniques 1157; 2023-04-13 14:54:16,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=9475, suffix heap size=148296,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=207 2023-04-13 14:54:16,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7591, totalHeapSize=17504, min timestamp=1681397656212, max timestamp=1681397656238Num uniques 1066; 2023-04-13 14:54:16,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48811, new segment data size=7591, suffix heap size=138832,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=166 2023-04-13 14:54:16,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7590, totalHeapSize=17504, min timestamp=1681397656212, max timestamp=1681397656238Num uniques 1066; 2023-04-13 14:54:16,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1681397656212, max timestamp=1681397656238Num uniques 1066; 2023-04-13 14:54:16,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48810, new segment data size=7590, suffix heap size=138832,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=166 2023-04-13 14:54:16,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=7596, suffix heap size=138832,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=166 2023-04-13 14:54:16,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=375, cellSize=17170, totalHeapSize=39240, min timestamp=1681397656242, max timestamp=1681397656256Num uniques 1025; 2023-04-13 14:54:16,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=17170, suffix heap size=134568,new segment heap size=39240 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=375 2023-04-13 14:54:16,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=375, cellSize=17170, totalHeapSize=39240, min timestamp=1681397656242, max timestamp=1681397656256Num uniques 1025; 2023-04-13 14:54:16,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=375, cellSize=17170, totalHeapSize=39240, min timestamp=1681397656242, max timestamp=1681397656256Num uniques 1025; 2023-04-13 14:54:16,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=17170, suffix heap size=134568,new segment heap size=39240 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=375 2023-04-13 14:54:16,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=17170, suffix heap size=134568,new segment heap size=39240 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=375 2023-04-13 14:54:16,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41142 deadline: 1681397716270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1681397716271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41122 deadline: 1681397716272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=434, cellSize=19874, totalHeapSize=45376, min timestamp=1681397656258, max timestamp=1681397656272Num uniques 1234; 2023-04-13 14:54:16,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=56514, new segment data size=19874, suffix heap size=156304,new segment heap size=45376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1234, new segment cells count=434 2023-04-13 14:54:16,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=434, cellSize=19874, totalHeapSize=45376, min timestamp=1681397656258, max timestamp=1681397656272Num uniques 1234; 2023-04-13 14:54:16,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=434, cellSize=19874, totalHeapSize=45376, min timestamp=1681397656258, max timestamp=1681397656272Num uniques 1234; 2023-04-13 14:54:16,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=56514, new segment data size=19874, suffix heap size=156304,new segment heap size=45376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1234, new segment cells count=434 2023-04-13 14:54:16,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=56514, new segment data size=19874, suffix heap size=156304,new segment heap size=45376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1234, new segment cells count=434 2023-04-13 14:54:16,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41140 deadline: 1681397716307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1681397656297, max timestamp=1681397656305Num uniques 1293; 2023-04-13 14:54:16,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1681397656297, max timestamp=1681397656305Num uniques 1293; 2023-04-13 14:54:16,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=59218, new segment data size=11128, suffix heap size=162440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1293, new segment cells count=243 2023-04-13 14:54:16,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1681397656297, max timestamp=1681397656305Num uniques 1293; 2023-04-13 14:54:16,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=59218, new segment data size=11128, suffix heap size=162440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1293, new segment cells count=243 2023-04-13 14:54:16,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=59218, new segment data size=11128, suffix heap size=162440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1293, new segment cells count=243 2023-04-13 14:54:16,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1681397656330, max timestamp=1681397656347Num uniques 1102; 2023-04-13 14:54:16,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=9250, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-04-13 14:54:16,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1681397656330, max timestamp=1681397656347Num uniques 1102; 2023-04-13 14:54:16,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=9250, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-04-13 14:54:16,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397656335, max timestamp=1681397656347Num uniques 1102; 2023-04-13 14:54:16,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6870, suffix heap size=142576,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=150 2023-04-13 14:54:16,376 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=10, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/207c612f0cfd4417940d69c45e3291d5 2023-04-13 14:54:16,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1681397656375, max timestamp=1681397656401Num uniques 1061; 2023-04-13 14:54:16,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41122 deadline: 1681397716399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=7366, suffix heap size=138312,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=161 2023-04-13 14:54:16,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1681397656375, max timestamp=1681397656401Num uniques 1009; 2023-04-13 14:54:16,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1681397656375, max timestamp=1681397656401Num uniques 1061; 2023-04-13 14:54:16,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=7366, suffix heap size=138312,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=161 2023-04-13 14:54:16,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46206, new segment data size=7366, suffix heap size=132904,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=161 2023-04-13 14:54:16,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1681397656407, max timestamp=1681397656418Num uniques 1020; 2023-04-13 14:54:16,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1681397656407, max timestamp=1681397656418Num uniques 1020; 2023-04-13 14:54:16,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=12360, suffix heap size=134048,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=270 2023-04-13 14:54:16,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=12360, suffix heap size=134048,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=270 2023-04-13 14:54:16,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1681397656407, max timestamp=1681397656418Num uniques 1020; 2023-04-13 14:54:16,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=12360, suffix heap size=134048,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=270 2023-04-13 14:54:16,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=379, cellSize=17354, totalHeapSize=39656, min timestamp=1681397656421, max timestamp=1681397656434Num uniques 1129; 2023-04-13 14:54:16,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51704, new segment data size=17354, suffix heap size=145384,new segment heap size=39656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=379 2023-04-13 14:54:16,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=379, cellSize=17354, totalHeapSize=39656, min timestamp=1681397656421, max timestamp=1681397656434Num uniques 1129; 2023-04-13 14:54:16,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=379, cellSize=17354, totalHeapSize=39656, min timestamp=1681397656421, max timestamp=1681397656434Num uniques 1129; 2023-04-13 14:54:16,437 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51704, new segment data size=17354, suffix heap size=145384,new segment heap size=39656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=379 2023-04-13 14:54:16,437 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51704, new segment data size=17354, suffix heap size=145384,new segment heap size=39656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=379 2023-04-13 14:54:16,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41140 deadline: 1681397716446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=288, cellSize=13188, totalHeapSize=30192, min timestamp=1681397656441, max timestamp=1681397656448Num uniques 1238; 2023-04-13 14:54:16,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41142 deadline: 1681397716449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=56698, new segment data size=13188, suffix heap size=156720,new segment heap size=30192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1238, new segment cells count=288 2023-04-13 14:54:16,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397656441, max timestamp=1681397656448Num uniques 1238; 2023-04-13 14:54:16,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=56698, new segment data size=6870, suffix heap size=156720,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1238, new segment cells count=150 2023-04-13 14:54:16,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=288, cellSize=13188, totalHeapSize=30192, min timestamp=1681397656441, max timestamp=1681397656448Num uniques 1238; 2023-04-13 14:54:16,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=56698, new segment data size=13188, suffix heap size=156720,new segment heap size=30192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1238, new segment cells count=288 2023-04-13 14:54:16,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=297, cellSize=13602, totalHeapSize=31128, min timestamp=1681397656456, max timestamp=1681397656467Num uniques 1147; 2023-04-13 14:54:16,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52532, new segment data size=13602, suffix heap size=147256,new segment heap size=31128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1147, new segment cells count=297 2023-04-13 14:54:16,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=297, cellSize=13602, totalHeapSize=31128, min timestamp=1681397656456, max timestamp=1681397656467Num uniques 1147; 2023-04-13 14:54:16,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52532, new segment data size=13602, suffix heap size=147256,new segment heap size=31128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1147, new segment cells count=297 2023-04-13 14:54:16,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=297, cellSize=13602, totalHeapSize=31128, min timestamp=1681397656456, max timestamp=1681397656467Num uniques 1009; 2023-04-13 14:54:16,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=13602, suffix heap size=132904,new segment heap size=31128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=297 2023-04-13 14:54:16,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397716496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1681397656480, max timestamp=1681397656496Num uniques 1156; 2023-04-13 14:54:16,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1681397656480, max timestamp=1681397656496Num uniques 1156; 2023-04-13 14:54:16,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1681397656480, max timestamp=1681397656496Num uniques 1156; 2023-04-13 14:54:16,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52940, new segment data size=7140, suffix heap size=148192,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1156, new segment cells count=156 2023-04-13 14:54:16,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52940, new segment data size=7140, suffix heap size=148192,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1156, new segment cells count=156 2023-04-13 14:54:16,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52940, new segment data size=7140, suffix heap size=148192,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1156, new segment cells count=156 2023-04-13 14:54:16,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1681397656520, max timestamp=1681397656525Num uniques 1015; 2023-04-13 14:54:16,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=7550, suffix heap size=133528,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=165 2023-04-13 14:54:16,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=265, cellSize=12130, totalHeapSize=27800, min timestamp=1681397656512, max timestamp=1681397656525Num uniques 1015; 2023-04-13 14:54:16,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=12130, suffix heap size=133528,new segment heap size=27800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=265 2023-04-13 14:54:16,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1681397656520, max timestamp=1681397656525Num uniques 1015; 2023-04-13 14:54:16,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=7550, suffix heap size=133528,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=165 2023-04-13 14:54:16,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1681397656532, max timestamp=1681397656553Num uniques 1024; 2023-04-13 14:54:16,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=10254, suffix heap size=134464,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=224 2023-04-13 14:54:16,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1681397656532, max timestamp=1681397656553Num uniques 1024; 2023-04-13 14:54:16,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=10254, suffix heap size=134464,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=224 2023-04-13 14:54:16,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1681397656532, max timestamp=1681397656553Num uniques 1124; 2023-04-13 14:54:16,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51474, new segment data size=10254, suffix heap size=144864,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1124, new segment cells count=224 2023-04-13 14:54:16,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8378, totalHeapSize=19272, min timestamp=1681397656587, max timestamp=1681397656590Num uniques 1083; 2023-04-13 14:54:16,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8378, totalHeapSize=19272, min timestamp=1681397656587, max timestamp=1681397656590Num uniques 1083; 2023-04-13 14:54:16,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=8378, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-04-13 14:54:16,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=8378, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-04-13 14:54:16,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8378, totalHeapSize=19272, min timestamp=1681397656587, max timestamp=1681397656590Num uniques 1083; 2023-04-13 14:54:16,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=8378, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-04-13 14:54:16,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=242, cellSize=11082, totalHeapSize=25408, min timestamp=1681397656620, max timestamp=1681397656625Num uniques 1042; 2023-04-13 14:54:16,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=11082, suffix heap size=136336,new segment heap size=25408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=242 2023-04-13 14:54:16,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=242, cellSize=11082, totalHeapSize=25408, min timestamp=1681397656620, max timestamp=1681397656625Num uniques 1042; 2023-04-13 14:54:16,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=11082, suffix heap size=136336,new segment heap size=25408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=242 2023-04-13 14:54:16,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=242, cellSize=11082, totalHeapSize=25408, min timestamp=1681397656620, max timestamp=1681397656625Num uniques 1042; 2023-04-13 14:54:16,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=11082, suffix heap size=136336,new segment heap size=25408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=242 2023-04-13 14:54:16,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=251, cellSize=11495, totalHeapSize=26344, min timestamp=1681397656635, max timestamp=1681397656640Num uniques 1101; 2023-04-13 14:54:16,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50425, new segment data size=11495, suffix heap size=142472,new segment heap size=26344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1101, new segment cells count=251 2023-04-13 14:54:16,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=251, cellSize=11495, totalHeapSize=26344, min timestamp=1681397656635, max timestamp=1681397656640Num uniques 1101; 2023-04-13 14:54:16,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=251, cellSize=11495, totalHeapSize=26344, min timestamp=1681397656635, max timestamp=1681397656640Num uniques 1101; 2023-04-13 14:54:16,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50425, new segment data size=11495, suffix heap size=142472,new segment heap size=26344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1101, new segment cells count=251 2023-04-13 14:54:16,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50425, new segment data size=11495, suffix heap size=142472,new segment heap size=26344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1101, new segment cells count=251 2023-04-13 14:54:16,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=210, cellSize=9610, totalHeapSize=22080, min timestamp=1681397656649, max timestamp=1681397656653Num uniques 1110; 2023-04-13 14:54:16,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50830, new segment data size=9610, suffix heap size=143408,new segment heap size=22080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1110, new segment cells count=210 2023-04-13 14:54:16,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=210, cellSize=9610, totalHeapSize=22080, min timestamp=1681397656649, max timestamp=1681397656653Num uniques 1110; 2023-04-13 14:54:16,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50830, new segment data size=9610, suffix heap size=143408,new segment heap size=22080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1110, new segment cells count=210 2023-04-13 14:54:16,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397656649, max timestamp=1681397656653Num uniques 1110; 2023-04-13 14:54:16,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50830, new segment data size=6870, suffix heap size=143408,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1110, new segment cells count=150 2023-04-13 14:54:16,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=169, cellSize=7734, totalHeapSize=17816, min timestamp=1681397656663, max timestamp=1681397656669Num uniques 1009; 2023-04-13 14:54:16,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=169, cellSize=7734, totalHeapSize=17816, min timestamp=1681397656663, max timestamp=1681397656669Num uniques 1069; 2023-04-13 14:54:16,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=169, cellSize=7734, totalHeapSize=17816, min timestamp=1681397656663, max timestamp=1681397656669Num uniques 1069; 2023-04-13 14:54:16,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48954, new segment data size=7734, suffix heap size=139144,new segment heap size=17816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1069, new segment cells count=169 2023-04-13 14:54:16,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=7734, suffix heap size=132904,new segment heap size=17816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=169 2023-04-13 14:54:16,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397716668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48954, new segment data size=7734, suffix heap size=139144,new segment heap size=17816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1069, new segment cells count=169 2023-04-13 14:54:16,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=178, cellSize=8148, totalHeapSize=18752, min timestamp=1681397656678, max timestamp=1681397656713Num uniques 1028; 2023-04-13 14:54:16,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47078, new segment data size=8148, suffix heap size=134880,new segment heap size=18752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1028, new segment cells count=178 2023-04-13 14:54:16,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=178, cellSize=8148, totalHeapSize=18752, min timestamp=1681397656678, max timestamp=1681397656713Num uniques 1028; 2023-04-13 14:54:16,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=178, cellSize=8148, totalHeapSize=18752, min timestamp=1681397656678, max timestamp=1681397656713Num uniques 1028; 2023-04-13 14:54:16,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47078, new segment data size=8148, suffix heap size=134880,new segment heap size=18752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1028, new segment cells count=178 2023-04-13 14:54:16,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47078, new segment data size=8148, suffix heap size=134880,new segment heap size=18752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1028, new segment cells count=178 2023-04-13 14:54:16,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=387, cellSize=17722, totalHeapSize=40488, min timestamp=1681397656723, max timestamp=1681397656732Num uniques 1037; 2023-04-13 14:54:16,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47492, new segment data size=17722, suffix heap size=135816,new segment heap size=40488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1037, new segment cells count=387 2023-04-13 14:54:16,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=387, cellSize=17722, totalHeapSize=40488, min timestamp=1681397656723, max timestamp=1681397656732Num uniques 1037; 2023-04-13 14:54:16,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=387, cellSize=17722, totalHeapSize=40488, min timestamp=1681397656723, max timestamp=1681397656732Num uniques 1037; 2023-04-13 14:54:16,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47492, new segment data size=17722, suffix heap size=135816,new segment heap size=40488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1037, new segment cells count=387 2023-04-13 14:54:16,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47492, new segment data size=17722, suffix heap size=135816,new segment heap size=40488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1037, new segment cells count=387 2023-04-13 14:54:16,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41140 deadline: 1681397716747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397716748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41122 deadline: 1681397716750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=196, cellSize=8976, totalHeapSize=20624, min timestamp=1681397656745, max timestamp=1681397656748Num uniques 1246; 2023-04-13 14:54:16,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=57066, new segment data size=8976, suffix heap size=157552,new segment heap size=20624 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1246, new segment cells count=196 2023-04-13 14:54:16,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=196, cellSize=8976, totalHeapSize=20624, min timestamp=1681397656745, max timestamp=1681397656748Num uniques 1246; 2023-04-13 14:54:16,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=57066, new segment data size=8976, suffix heap size=157552,new segment heap size=20624 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1246, new segment cells count=196 2023-04-13 14:54:16,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=196, cellSize=8976, totalHeapSize=20624, min timestamp=1681397656745, max timestamp=1681397656748Num uniques 1246; 2023-04-13 14:54:16,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=57066, new segment data size=8976, suffix heap size=157552,new segment heap size=20624 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1246, new segment cells count=196 2023-04-13 14:54:16,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=255, cellSize=11675, totalHeapSize=26760, min timestamp=1681397656777, max timestamp=1681397656783Num uniques 1055; 2023-04-13 14:54:16,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48315, new segment data size=11675, suffix heap size=137688,new segment heap size=26760 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1055, new segment cells count=255 2023-04-13 14:54:16,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=255, cellSize=11675, totalHeapSize=26760, min timestamp=1681397656777, max timestamp=1681397656783Num uniques 1055; 2023-04-13 14:54:16,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48315, new segment data size=11675, suffix heap size=137688,new segment heap size=26760 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1055, new segment cells count=255 2023-04-13 14:54:16,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=255, cellSize=11675, totalHeapSize=26760, min timestamp=1681397656777, max timestamp=1681397656783Num uniques 1055; 2023-04-13 14:54:16,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48315, new segment data size=11675, suffix heap size=137688,new segment heap size=26760 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1055, new segment cells count=255 2023-04-13 14:54:16,788 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/02a21700016c416c9b6c7a4879afb23f 2023-04-13 14:54:16,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=214, cellSize=9794, totalHeapSize=22496, min timestamp=1681397656803, max timestamp=1681397656825Num uniques 1114; 2023-04-13 14:54:16,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51014, new segment data size=9794, suffix heap size=143824,new segment heap size=22496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1114, new segment cells count=214 2023-04-13 14:54:16,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=214, cellSize=9794, totalHeapSize=22496, min timestamp=1681397656803, max timestamp=1681397656825Num uniques 1114; 2023-04-13 14:54:16,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51014, new segment data size=9794, suffix heap size=143824,new segment heap size=22496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1114, new segment cells count=214 2023-04-13 14:54:16,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=214, cellSize=9794, totalHeapSize=22496, min timestamp=1681397656803, max timestamp=1681397656825Num uniques 1114; 2023-04-13 14:54:16,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51014, new segment data size=9794, suffix heap size=143824,new segment heap size=22496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1114, new segment cells count=214 2023-04-13 14:54:16,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=173, cellSize=7914, totalHeapSize=18232, min timestamp=1681397656852, max timestamp=1681397656856Num uniques 1073; 2023-04-13 14:54:16,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49134, new segment data size=7914, suffix heap size=139560,new segment heap size=18232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1073, new segment cells count=173 2023-04-13 14:54:16,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=173, cellSize=7914, totalHeapSize=18232, min timestamp=1681397656852, max timestamp=1681397656856Num uniques 1073; 2023-04-13 14:54:16,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=173, cellSize=7911, totalHeapSize=18232, min timestamp=1681397656852, max timestamp=1681397656856Num uniques 1073; 2023-04-13 14:54:16,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49134, new segment data size=7914, suffix heap size=139560,new segment heap size=18232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1073, new segment cells count=173 2023-04-13 14:54:16,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49131, new segment data size=7911, suffix heap size=139560,new segment heap size=18232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1073, new segment cells count=173 2023-04-13 14:54:16,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=282, cellSize=12912, totalHeapSize=29568, min timestamp=1681397656869, max timestamp=1681397656876Num uniques 1032; 2023-04-13 14:54:16,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=282, cellSize=12912, totalHeapSize=29568, min timestamp=1681397656869, max timestamp=1681397656876Num uniques 1032; 2023-04-13 14:54:16,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=282, cellSize=12912, totalHeapSize=29568, min timestamp=1681397656869, max timestamp=1681397656876Num uniques 1032; 2023-04-13 14:54:16,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47262, new segment data size=12912, suffix heap size=135296,new segment heap size=29568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1032, new segment cells count=282 2023-04-13 14:54:16,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47262, new segment data size=12912, suffix heap size=135296,new segment heap size=29568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1032, new segment cells count=282 2023-04-13 14:54:16,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47262, new segment data size=12912, suffix heap size=135296,new segment heap size=29568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1032, new segment cells count=282 2023-04-13 14:54:16,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=391, cellSize=17906, totalHeapSize=40904, min timestamp=1681397656884, max timestamp=1681397656894Num uniques 1141; 2023-04-13 14:54:16,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52256, new segment data size=17906, suffix heap size=146632,new segment heap size=40904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1141, new segment cells count=391 2023-04-13 14:54:16,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=391, cellSize=17906, totalHeapSize=40904, min timestamp=1681397656884, max timestamp=1681397656894Num uniques 1141; 2023-04-13 14:54:16,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52256, new segment data size=17906, suffix heap size=146632,new segment heap size=40904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1141, new segment cells count=391 2023-04-13 14:54:16,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=391, cellSize=17906, totalHeapSize=40904, min timestamp=1681397656884, max timestamp=1681397656894Num uniques 1141; 2023-04-13 14:54:16,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52256, new segment data size=17906, suffix heap size=146632,new segment heap size=40904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1141, new segment cells count=391 2023-04-13 14:54:16,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397716912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397716913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:16,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41122 deadline: 1681397716913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:16,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=250, cellSize=11450, totalHeapSize=26240, min timestamp=1681397656909, max timestamp=1681397656916Num uniques 1250; 2023-04-13 14:54:16,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=57250, new segment data size=11450, suffix heap size=157968,new segment heap size=26240 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1250, new segment cells count=250 2023-04-13 14:54:16,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=250, cellSize=11450, totalHeapSize=26240, min timestamp=1681397656909, max timestamp=1681397656916Num uniques 1250; 2023-04-13 14:54:16,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=57250, new segment data size=11450, suffix heap size=157968,new segment heap size=26240 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1250, new segment cells count=250 2023-04-13 14:54:16,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=250, cellSize=11450, totalHeapSize=26240, min timestamp=1681397656909, max timestamp=1681397656916Num uniques 1250; 2023-04-13 14:54:16,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=57250, new segment data size=11450, suffix heap size=157968,new segment heap size=26240 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1250, new segment cells count=250 2023-04-13 14:54:16,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397656943, max timestamp=1681397656955Num uniques 1109; 2023-04-13 14:54:16,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50785, new segment data size=9565, suffix heap size=143304,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1109, new segment cells count=209 2023-04-13 14:54:16,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397656943, max timestamp=1681397656955Num uniques 1109; 2023-04-13 14:54:16,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50785, new segment data size=9565, suffix heap size=143304,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1109, new segment cells count=209 2023-04-13 14:54:16,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397656943, max timestamp=1681397656955Num uniques 1109; 2023-04-13 14:54:16,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50785, new segment data size=6870, suffix heap size=143304,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1109, new segment cells count=150 2023-04-13 14:54:16,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397656979, max timestamp=1681397656992Num uniques 1009; 2023-04-13 14:54:16,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397656979, max timestamp=1681397656992Num uniques 1068; 2023-04-13 14:54:16,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397656979, max timestamp=1681397656992Num uniques 1068; 2023-04-13 14:54:16,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-13 14:54:16,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-13 14:54:16,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-13 14:54:17,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681397657015, max timestamp=1681397657024Num uniques 1027; 2023-04-13 14:54:17,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 14:54:17,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397657015, max timestamp=1681397657024Num uniques 1027; 2023-04-13 14:54:17,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-13 14:54:17,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397657015, max timestamp=1681397657024Num uniques 1027; 2023-04-13 14:54:17,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-13 14:54:17,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=536, cellSize=24546, totalHeapSize=55984, min timestamp=1681397657033, max timestamp=1681397657042Num uniques 1036; 2023-04-13 14:54:17,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=24546, suffix heap size=135712,new segment heap size=55984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=536 2023-04-13 14:54:17,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=536, cellSize=24546, totalHeapSize=55984, min timestamp=1681397657033, max timestamp=1681397657042Num uniques 1009; 2023-04-13 14:54:17,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=24546, suffix heap size=132904,new segment heap size=55984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=536 2023-04-13 14:54:17,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=536, cellSize=24546, totalHeapSize=55984, min timestamp=1681397657033, max timestamp=1681397657042Num uniques 1009; 2023-04-13 14:54:17,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=24546, suffix heap size=132904,new segment heap size=55984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=536 2023-04-13 14:54:17,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397717056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397717057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41140 deadline: 1681397717060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397717061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41122 deadline: 1681397717061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397717163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41140 deadline: 1681397717163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397717163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41122 deadline: 1681397717166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397717167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,239 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/19f01c9600ea49f8b800e4ae25590d0f 2023-04-13 14:54:17,245 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/207c612f0cfd4417940d69c45e3291d5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/207c612f0cfd4417940d69c45e3291d5 2023-04-13 14:54:17,250 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/207c612f0cfd4417940d69c45e3291d5, entries=100, sequenceid=10, filesize=18.4 K 2023-04-13 14:54:17,251 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/02a21700016c416c9b6c7a4879afb23f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/02a21700016c416c9b6c7a4879afb23f 2023-04-13 14:54:17,259 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/02a21700016c416c9b6c7a4879afb23f, entries=100, sequenceid=10, filesize=9.3 K 2023-04-13 14:54:17,259 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/19f01c9600ea49f8b800e4ae25590d0f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/19f01c9600ea49f8b800e4ae25590d0f 2023-04-13 14:54:17,270 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/19f01c9600ea49f8b800e4ae25590d0f, entries=100, sequenceid=10, filesize=9.3 K 2023-04-13 14:54:17,271 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=181.14 KB/185490 for 0b6ed6780bbc11018cb686e788422bc1 in 1778ms, sequenceid=10, compaction requested=false 2023-04-13 14:54:17,271 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:17,271 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:17,271 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:17,271 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:17,271 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,271 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:17,271 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:17,274 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,274 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,275 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,275 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:17,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:17,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cf78fd6[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:17,277 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:17,277 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,278 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,278 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:17,278 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:17,278 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:17,278 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,278 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:17,279 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:17,279 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:17,279 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,279 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:17,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:17,282 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:17,283 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:17,283 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:17,283 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:17,283 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:17,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:17,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,287 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,288 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,293 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:17,293 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:17,293 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,293 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:17,293 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:17,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:17,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:17,293 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,293 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,293 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,293 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,293 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:17,293 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:17,293 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:17,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:17,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:17,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:17,370 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=194.56 KB heapSize=529.64 KB 2023-04-13 14:54:17,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,370 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:17,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41140 deadline: 1681397717367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,370 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:54:17,370 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:17,370 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:54:17,370 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:17,370 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:54:17,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397717371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397717372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397717372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41122 deadline: 1681397717372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,393 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:17,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:17,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:17,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:17,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:17,402 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:17,402 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:17,402 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:17,402 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:17,403 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,403 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:17,404 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:17,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:17,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,405 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,405 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:17,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:17,406 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413b59cd31294664d85ba061c164125ca36 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b59cd31294664d85ba061c164125ca36 2023-04-13 14:54:17,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:17,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:17,408 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:17,408 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:17,408 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:17,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:17,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:17,412 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:17,412 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:17,412 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:17,412 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:17,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:17,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:17,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,422 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:17,422 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:17,423 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,423 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,424 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,424 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:17,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:17,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bc3e9f2[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:17,426 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:17,426 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,428 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,428 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:17,428 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:17,428 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:17,428 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,429 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:17,430 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,430 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:17,430 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:17,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,430 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:17,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:17,435 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:17,435 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:17,435 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:17,435 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:17,435 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:17,438 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,438 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:17,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,439 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,440 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,442 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,448 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:17,448 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:17,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:17,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:17,448 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,448 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,448 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:17,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:17,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:17,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:17,448 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=959, memsize=64.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/02c4316e314b4a9396f612105c6b83e2 2023-04-13 14:54:17,448 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,449 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,449 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:17,449 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,449 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,449 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,449 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:17,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:17,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:17,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397717476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397717477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,549 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:17,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:17,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:17,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:17,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:17,553 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:17,553 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:17,553 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:17,553 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:17,555 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,555 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,557 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:17,557 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:17,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:17,557 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,557 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:17,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:17,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:17,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:17,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:17,559 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:17,559 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:17,559 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:17,559 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:17,559 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:17,559 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:17,559 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:17,559 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:17,559 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:17,559 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:17,559 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:17,559 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,560 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:17,560 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:17,561 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,561 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,562 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,562 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:17,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f835b8a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:17,566 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:17,566 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,569 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,569 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,569 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:17,569 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:17,569 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:17,569 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:17,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:17,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:17,572 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:17,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:17,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:17,576 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:17,577 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:17,577 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:17,577 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:17,577 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:17,579 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,579 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:17,579 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:17,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:17,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,581 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,584 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:17,584 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:17,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:17,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:17,584 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,584 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,584 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,584 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,584 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,585 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:17,584 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:17,585 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:17,585 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:17,584 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:17,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:17,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397717672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41140 deadline: 1681397717676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41122 deadline: 1681397717679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397717680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,685 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:17,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:17,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:17,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397717681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:17,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:17,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:17,690 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:17,690 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:17,690 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:17,690 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:17,690 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,690 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,692 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:17,692 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:17,693 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,693 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:17,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:17,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:17,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:17,694 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:17,694 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:17,694 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:17,694 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:17,694 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:17,694 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:17,694 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:17,694 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:17,694 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:17,694 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:17,694 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:17,694 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,694 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:17,694 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:17,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,696 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:17,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:17,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29f65bd8[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:17,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,700 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,700 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:17,700 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:17,700 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:17,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,701 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:17,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:17,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:17,702 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:17,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:17,704 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:17,704 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:17,704 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:17,704 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:17,704 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:17,707 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,707 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,707 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:17,708 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,710 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 14:54:17,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,716 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,720 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:17,720 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:17,720 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:17,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:17,720 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,720 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,720 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,720 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,720 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,720 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:17,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:17,728 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:17,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:17,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:17,728 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:17,728 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:17,828 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:17,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:17,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:17,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:17,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:17,836 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:17,836 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:17,836 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:17,836 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:17,837 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,837 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,838 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:17,838 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:17,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:17,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:17,840 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,840 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:17,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:17,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:17,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:17,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:17,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:17,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:17,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:17,841 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:17,841 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:17,841 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:17,841 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:17,842 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,842 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,843 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,843 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:17,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69bfe80f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:17,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:17,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,847 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:17,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:17,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:17,847 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,847 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:17,848 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,849 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:17,849 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:17,849 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:17,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:17,853 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:17,853 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:17,853 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:17,853 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:17,853 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:17,855 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,855 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,855 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:17,856 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,858 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,861 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,861 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:17,861 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,861 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,862 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,862 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,862 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,862 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:17,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:17,861 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:17,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:17,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:17,862 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:17,862 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:17,862 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:17,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:17,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:17,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:17,870 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=959 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/17e14893526744d29dcf5eb067e28254 2023-04-13 14:54:17,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=959 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/73e0cde10bb748c5a49f99a365bb55f8 2023-04-13 14:54:17,913 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/02c4316e314b4a9396f612105c6b83e2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/02c4316e314b4a9396f612105c6b83e2 2023-04-13 14:54:17,919 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/02c4316e314b4a9396f612105c6b83e2, entries=150, sequenceid=959, filesize=25.4 K 2023-04-13 14:54:17,920 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/17e14893526744d29dcf5eb067e28254 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/17e14893526744d29dcf5eb067e28254 2023-04-13 14:54:17,932 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/17e14893526744d29dcf5eb067e28254, entries=150, sequenceid=959, filesize=11.9 K 2023-04-13 14:54:17,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/73e0cde10bb748c5a49f99a365bb55f8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/73e0cde10bb748c5a49f99a365bb55f8 2023-04-13 14:54:17,952 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/73e0cde10bb748c5a49f99a365bb55f8, entries=150, sequenceid=959, filesize=11.9 K 2023-04-13 14:54:17,953 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~194.56 KB/199230, heapSize ~529.59 KB/542304, currentSize=0 B/0 for 0b6ed6780bbc11018cb686e788422bc1 in 583ms, sequenceid=959, compaction requested=false 2023-04-13 14:54:17,953 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:17,962 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:17,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:17,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:17,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:17,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:17,977 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:17,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:17,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:17,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:17,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,982 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,982 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:17,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:17,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:17,983 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,983 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:17,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:17,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:17,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:17,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:17,987 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:17,987 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:17,987 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:17,987 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:17,987 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:17,988 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:17,988 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:17,988 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:17,988 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:17,988 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:17,988 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:17,988 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,988 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:17,988 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:17,990 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,990 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,990 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:17,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:17,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d1700b3[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:17,992 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:17,992 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,993 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:17,994 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:17,994 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:17,994 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:17,994 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,994 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:17,996 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,996 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:17,996 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:17,996 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:17,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:17,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:18,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:18,000 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:18,000 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:18,000 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:18,000 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:18,000 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:18,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:18,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:18,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:18,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:18,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:18,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:18,003 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:18,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,005 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,007 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,007 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:18,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:18,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:18,007 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:18,007 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:18,008 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,007 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:18,008 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,008 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:18,008 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:18,008 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,008 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,008 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:18,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:18,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:18,074 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:54:18,074 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:18,074 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:18,074 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:18,074 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:18,074 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:18,074 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:18,108 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:18,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:18,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:18,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:18,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:18,116 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:18,116 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:18,116 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:18,116 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:18,117 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,117 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,117 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304139ef3a22f53d04bc782999d9c11e5c027 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304139ef3a22f53d04bc782999d9c11e5c027 2023-04-13 14:54:18,118 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,118 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:18,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:18,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:18,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:18,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:18,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:18,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:18,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:18,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397658103, max timestamp=1681397658132Num uniques 859; 2023-04-13 14:54:18,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397658103, max timestamp=1681397658132Num uniques 859; 2023-04-13 14:54:18,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:18,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:18,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397658103, max timestamp=1681397658132Num uniques 859; 2023-04-13 14:54:18,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:18,152 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:18,152 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:18,152 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:18,152 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:18,152 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:18,152 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:18,152 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:18,152 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:18,152 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,152 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:18,153 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:18,153 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=970, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/91bc96a269b54e1f9d0622b13e4fde8f 2023-04-13 14:54:18,156 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,156 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:18,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:18,157 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:18,157 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:18,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:18,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:18,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:18,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28ad4d90[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:18,160 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:18,160 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,162 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,162 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:18,162 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:18,162 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:18,163 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:18,164 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:18,164 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:18,165 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,165 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:18,165 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:18,165 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:18,165 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:18,166 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:18,166 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:18,166 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:18,169 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,169 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:18,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,170 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:18,171 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:18,174 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,179 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:18,179 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:18,179 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,179 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,179 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,179 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,179 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,179 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,179 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:18,181 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:18,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:18,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:18,181 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:18,181 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:18,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:18,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:18,184 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=970 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/a7b8c9d9041947b4ad4120b524bb7970 2023-04-13 14:54:18,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41140 deadline: 1681397718181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41122 deadline: 1681397718184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397718185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397718186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397718187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,281 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:18,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:18,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:18,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:18,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:18,285 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:18,285 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:18,285 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:18,285 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:18,286 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,286 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,287 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:18,287 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:18,287 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,288 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:18,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:18,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:18,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:18,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:18,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:18,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:18,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:18,289 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:18,289 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:18,289 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:18,289 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:18,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:18,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:18,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:18,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,291 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:18,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397718291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397718291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397718292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:18,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@331efbc6[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:18,294 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:18,294 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,295 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,295 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:18,295 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:18,295 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:18,295 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,295 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:18,297 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:18,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:18,298 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:18,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:18,300 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:18,300 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:18,300 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:18,300 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:18,300 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:18,302 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,302 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:18,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,302 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:18,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,303 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,305 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,307 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:18,307 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:18,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:18,307 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,307 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,307 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,307 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,307 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,307 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,307 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,308 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:18,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:18,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:18,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:18,308 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:18,308 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:18,408 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:18,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:18,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:18,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:18,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:18,412 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:18,412 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:18,412 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:18,412 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:18,413 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,413 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,414 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,414 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:18,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:18,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:18,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:18,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:18,416 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:18,416 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:18,416 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:18,416 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:18,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,421 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:18,421 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,421 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:18,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:18,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25f7823a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:18,423 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:18,423 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,424 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,424 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:18,424 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,424 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:18,424 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:18,424 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:18,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,425 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:18,426 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:18,426 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:18,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:18,428 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:18,428 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:18,428 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:18,428 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:18,428 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:18,430 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,430 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,430 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:18,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:18,431 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,432 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,435 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,435 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,435 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,435 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,435 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:18,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:18,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:18,435 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,435 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:18,435 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:18,435 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:18,435 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:18,435 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,436 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:18,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:18,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397718495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397718495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397718496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,536 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:18,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:18,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:18,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:18,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:18,539 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:18,539 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:18,539 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:18,540 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:18,540 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,540 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,541 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:18,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:18,541 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,542 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:18,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:18,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:18,544 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:18,544 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:18,544 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:18,544 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:18,546 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,546 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:18,546 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,546 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:18,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:18,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50901caf[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:18,548 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:18,548 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,549 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,549 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:18,549 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:18,549 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:18,549 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,549 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:18,551 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,551 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:18,551 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:18,551 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:18,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:18,554 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:18,556 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:18,556 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:18,556 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:18,556 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:18,559 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,559 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,559 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:18,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:18,560 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,561 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,563 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,563 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:18,563 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:18,564 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:18,564 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:18,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:18,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:18,564 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,564 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,564 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,564 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:18,564 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:18,564 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:18,564 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:18,606 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=970 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/0425b9852b24474ab5410a84a13f7d8c 2023-04-13 14:54:18,612 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/91bc96a269b54e1f9d0622b13e4fde8f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/91bc96a269b54e1f9d0622b13e4fde8f 2023-04-13 14:54:18,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/91bc96a269b54e1f9d0622b13e4fde8f, entries=350, sequenceid=970, filesize=52.9 K 2023-04-13 14:54:18,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/a7b8c9d9041947b4ad4120b524bb7970 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/a7b8c9d9041947b4ad4120b524bb7970 2023-04-13 14:54:18,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/a7b8c9d9041947b4ad4120b524bb7970, entries=150, sequenceid=970, filesize=11.9 K 2023-04-13 14:54:18,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/0425b9852b24474ab5410a84a13f7d8c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0425b9852b24474ab5410a84a13f7d8c 2023-04-13 14:54:18,634 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0425b9852b24474ab5410a84a13f7d8c, entries=150, sequenceid=970, filesize=11.9 K 2023-04-13 14:54:18,635 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 0b6ed6780bbc11018cb686e788422bc1 in 562ms, sequenceid=970, compaction requested=true 2023-04-13 14:54:18,635 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:18,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:18,635 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:18,635 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:18,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:18,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:18,637 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 99058 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:18,637 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0b6ed6780bbc11018cb686e788422bc1/A is initiating minor compaction (all files) 2023-04-13 14:54:18,637 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b6ed6780bbc11018cb686e788422bc1/A in TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:18,637 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/207c612f0cfd4417940d69c45e3291d5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/02c4316e314b4a9396f612105c6b83e2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/91bc96a269b54e1f9d0622b13e4fde8f] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp, totalSize=96.7 K 2023-04-13 14:54:18,637 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33914 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:18,637 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0b6ed6780bbc11018cb686e788422bc1/B is initiating minor compaction (all files) 2023-04-13 14:54:18,637 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b6ed6780bbc11018cb686e788422bc1/B in TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:18,637 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/02a21700016c416c9b6c7a4879afb23f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/17e14893526744d29dcf5eb067e28254, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/a7b8c9d9041947b4ad4120b524bb7970] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp, totalSize=33.1 K 2023-04-13 14:54:18,638 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 207c612f0cfd4417940d69c45e3291d5, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681397655485 2023-04-13 14:54:18,639 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 02a21700016c416c9b6c7a4879afb23f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681397655485 2023-04-13 14:54:18,639 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 02c4316e314b4a9396f612105c6b83e2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=959, earliestPutTs=1681397657055 2023-04-13 14:54:18,639 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 17e14893526744d29dcf5eb067e28254, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=959, earliestPutTs=1681397657055 2023-04-13 14:54:18,640 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 91bc96a269b54e1f9d0622b13e4fde8f, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=970, earliestPutTs=1681397658052 2023-04-13 14:54:18,640 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting a7b8c9d9041947b4ad4120b524bb7970, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=970, earliestPutTs=1681397658065 2023-04-13 14:54:18,653 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b6ed6780bbc11018cb686e788422bc1#B#compaction#120 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:18,664 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:18,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:18,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:18,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:18,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:18,672 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:18,672 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:18,672 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:18,672 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:18,674 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b6ed6780bbc11018cb686e788422bc1#A#compaction#119 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:18,674 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,675 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,677 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:18,677 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:18,678 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,678 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:18,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:18,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:18,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:18,697 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:18,697 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:18,697 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:18,697 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:18,698 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-13 14:54:18,698 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:18,698 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:18,698 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:18,698 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:18,698 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:18,698 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:18,706 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/7882b81f765642bfb6e2789220ad01da as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/7882b81f765642bfb6e2789220ad01da 2023-04-13 14:54:18,714 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b6ed6780bbc11018cb686e788422bc1/B of 0b6ed6780bbc11018cb686e788422bc1 into 7882b81f765642bfb6e2789220ad01da(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:18,714 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:18,714 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., storeName=0b6ed6780bbc11018cb686e788422bc1/B, priority=13, startTime=1681397658635; duration=0sec 2023-04-13 14:54:18,715 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:18,715 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:18,717 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33914 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:18,717 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0b6ed6780bbc11018cb686e788422bc1/C is initiating minor compaction (all files) 2023-04-13 14:54:18,717 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b6ed6780bbc11018cb686e788422bc1/C in TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:18,717 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/19f01c9600ea49f8b800e4ae25590d0f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/73e0cde10bb748c5a49f99a365bb55f8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0425b9852b24474ab5410a84a13f7d8c] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp, totalSize=33.1 K 2023-04-13 14:54:18,718 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 19f01c9600ea49f8b800e4ae25590d0f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681397655485 2023-04-13 14:54:18,718 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 73e0cde10bb748c5a49f99a365bb55f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=959, earliestPutTs=1681397657055 2023-04-13 14:54:18,719 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 0425b9852b24474ab5410a84a13f7d8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=970, earliestPutTs=1681397658065 2023-04-13 14:54:18,754 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b6ed6780bbc11018cb686e788422bc1#C#compaction#122 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:18,776 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/32066e63404641b78e2341df8602a263 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/32066e63404641b78e2341df8602a263 2023-04-13 14:54:18,784 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b6ed6780bbc11018cb686e788422bc1/C of 0b6ed6780bbc11018cb686e788422bc1 into 32066e63404641b78e2341df8602a263(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:18,784 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:18,784 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., storeName=0b6ed6780bbc11018cb686e788422bc1/C, priority=13, startTime=1681397658635; duration=0sec 2023-04-13 14:54:18,784 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:18,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:18,803 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:18,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397718813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397718815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397718815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397718919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397718920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:18,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:18,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397718921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:19,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:19,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397719123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:19,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:19,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397719123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:19,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:19,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397719125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:19,159 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041350e12dee1708494abc562727ce5f845d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041350e12dee1708494abc562727ce5f845d 2023-04-13 14:54:19,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:19,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41140 deadline: 1681397719193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:19,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:19,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41122 deadline: 1681397719197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:19,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:19,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397719426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:19,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:19,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397719427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:19,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:19,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397719430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:19,540 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/55c72473efd949b6955de42567a861ce as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/55c72473efd949b6955de42567a861ce 2023-04-13 14:54:19,546 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b6ed6780bbc11018cb686e788422bc1/A of 0b6ed6780bbc11018cb686e788422bc1 into 55c72473efd949b6955de42567a861ce(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:19,546 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:19,546 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., storeName=0b6ed6780bbc11018cb686e788422bc1/A, priority=13, startTime=1681397658635; duration=0sec 2023-04-13 14:54:19,546 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:19,567 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1007, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/88f993cdb8bf41e18685fb8d51663b27 2023-04-13 14:54:19,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:19,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397719932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:19,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:19,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397719932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:19,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:19,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397719933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:19,998 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1007 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/d070d8b6feaa4058a22841ad42a9b8bc 2023-04-13 14:54:20,012 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1007 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/911cea0cca364293b05af43812b92ab2 2023-04-13 14:54:20,018 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/88f993cdb8bf41e18685fb8d51663b27 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/88f993cdb8bf41e18685fb8d51663b27 2023-04-13 14:54:20,032 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/88f993cdb8bf41e18685fb8d51663b27, entries=150, sequenceid=1007, filesize=25.4 K 2023-04-13 14:54:20,034 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/d070d8b6feaa4058a22841ad42a9b8bc as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/d070d8b6feaa4058a22841ad42a9b8bc 2023-04-13 14:54:20,039 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/d070d8b6feaa4058a22841ad42a9b8bc, entries=150, sequenceid=1007, filesize=11.9 K 2023-04-13 14:54:20,045 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/911cea0cca364293b05af43812b92ab2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/911cea0cca364293b05af43812b92ab2 2023-04-13 14:54:20,065 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/911cea0cca364293b05af43812b92ab2, entries=150, sequenceid=1007, filesize=11.9 K 2023-04-13 14:54:20,066 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 0b6ed6780bbc11018cb686e788422bc1 in 1369ms, sequenceid=1007, compaction requested=false 2023-04-13 14:54:20,066 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:20,066 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:20,066 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:20,066 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:20,066 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,072 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:20,072 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:20,075 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,075 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:20,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:20,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12130ae0[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:20,079 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:20,079 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:20,080 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:20,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:20,080 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:20,080 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:20,080 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:20,081 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,081 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:20,082 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:20,082 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:20,082 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,082 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:20,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:20,085 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:20,085 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:20,085 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:20,085 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:20,085 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:20,087 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:20,087 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:20,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:20,087 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:20,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:20,088 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,091 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,096 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:20,096 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:20,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:20,096 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:20,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:20,096 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:20,096 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:20,096 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,096 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:20,096 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:20,096 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:20,096 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:20,096 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:20,097 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:20,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:20,197 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:20,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:20,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:20,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:20,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:20,200 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:20,200 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:20,200 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:20,200 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:20,201 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:20,201 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:20,202 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,202 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:20,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:20,203 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:20,203 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:20,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:20,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:20,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:20,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:20,204 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:20,204 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:20,205 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 14:54:20,205 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:20,205 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:20,205 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:20,205 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:20,205 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:20,205 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:20,622 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041370390f46de1f407b8c1a010cc70c7ad0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041370390f46de1f407b8c1a010cc70c7ad0 2023-04-13 14:54:20,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:20,940 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:20,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397660956, max timestamp=1681397660964Num uniques 859; 2023-04-13 14:54:20,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:20,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397660956, max timestamp=1681397660965Num uniques 859; 2023-04-13 14:54:20,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:20,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397660956, max timestamp=1681397660965Num uniques 859; 2023-04-13 14:54:20,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:54:20,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681397660986, max timestamp=1681397660992Num uniques 1068; 2023-04-13 14:54:20,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-13 14:54:20,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681397660986, max timestamp=1681397660992Num uniques 1068; 2023-04-13 14:54:20,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-13 14:54:20,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681397660986, max timestamp=1681397660992Num uniques 1018; 2023-04-13 14:54:20,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-13 14:54:21,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:21,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397721019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:21,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397721020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:21,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397721023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,031 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1019, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/b25a91f98dac4d9c8a0292b5f01d5360 2023-04-13 14:54:21,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:21,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397721124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:21,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397721124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:21,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397721129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:21,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41140 deadline: 1681397721201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,204 DEBUG [Thread-847] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:54:21,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:21,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41122 deadline: 1681397721211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,214 DEBUG [Thread-843] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:54:21,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:21,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397721330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:21,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397721331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:21,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397721331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,452 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=1019 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/2cc23bc72cfd4350b9267919d17b6733 2023-04-13 14:54:21,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:21,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397721636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:21,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397721637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:21,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397721637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,863 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=1019 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/d4d2acd59cf24228925d7ba9a42d3ea1 2023-04-13 14:54:21,868 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/b25a91f98dac4d9c8a0292b5f01d5360 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/b25a91f98dac4d9c8a0292b5f01d5360 2023-04-13 14:54:21,872 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/b25a91f98dac4d9c8a0292b5f01d5360, entries=150, sequenceid=1019, filesize=25.4 K 2023-04-13 14:54:21,873 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/2cc23bc72cfd4350b9267919d17b6733 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/2cc23bc72cfd4350b9267919d17b6733 2023-04-13 14:54:21,878 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/2cc23bc72cfd4350b9267919d17b6733, entries=150, sequenceid=1019, filesize=11.9 K 2023-04-13 14:54:21,879 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/d4d2acd59cf24228925d7ba9a42d3ea1 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/d4d2acd59cf24228925d7ba9a42d3ea1 2023-04-13 14:54:21,883 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/d4d2acd59cf24228925d7ba9a42d3ea1, entries=150, sequenceid=1019, filesize=11.9 K 2023-04-13 14:54:21,884 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 0b6ed6780bbc11018cb686e788422bc1 in 1679ms, sequenceid=1019, compaction requested=true 2023-04-13 14:54:21,884 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:21,884 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:21,885 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:21,886 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:21,886 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:21,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:21,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@569713af[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:21,887 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:21,887 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:21,888 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:21,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:21,888 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:21,889 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:21,889 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:21,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:21,893 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:21,893 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,893 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:21,893 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:21,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:21,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:21,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:21,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:21,896 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:21,896 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:21,896 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:21,896 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:21,896 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:21,898 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:21,898 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:21,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:21,898 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:21,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:21,899 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,900 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,902 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,902 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:21,902 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:21,902 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:21,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:21,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:21,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:21,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:21,902 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:21,902 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:21,902 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:21,902 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:21,902 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:21,902 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:21,902 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:21,903 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:21,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:22,003 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:22,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:22,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:22,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:22,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:22,006 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:22,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:22,007 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:22,007 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:22,007 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:22,007 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:22,008 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:22,008 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:22,008 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,009 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:22,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:22,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:22,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:22,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:22,010 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:22,010 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:22,010 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:22,010 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:22,010 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:22,010 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:22,010 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:22,010 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=140.89 KB heapSize=394.69 KB 2023-04-13 14:54:22,010 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:22,010 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:22,010 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:22,010 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:22,010 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:22,010 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:22,028 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304137bc06edc695b4fd29af4e1974f9e0efc to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137bc06edc695b4fd29af4e1974f9e0efc 2023-04-13 14:54:22,034 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1072, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/455ff6ca5b414298966ff23a38f44375 2023-04-13 14:54:22,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:22,141 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:22,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:22,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397722147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:22,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397722148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:22,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397722149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:22,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397722250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:22,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397722251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:22,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397722253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:22,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397722451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:22,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397722453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,459 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1072 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/a8c58d644e0e4b599f3b03f6abb1ac32 2023-04-13 14:54:22,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:22,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397722459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:22,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397722757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:22,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397722758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:22,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397722764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,871 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1072 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/5cbef882dc52454ba07ab8c7e43466da 2023-04-13 14:54:22,877 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/455ff6ca5b414298966ff23a38f44375 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/455ff6ca5b414298966ff23a38f44375 2023-04-13 14:54:22,895 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/455ff6ca5b414298966ff23a38f44375, entries=150, sequenceid=1072, filesize=25.4 K 2023-04-13 14:54:22,896 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/a8c58d644e0e4b599f3b03f6abb1ac32 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/a8c58d644e0e4b599f3b03f6abb1ac32 2023-04-13 14:54:22,900 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/a8c58d644e0e4b599f3b03f6abb1ac32, entries=150, sequenceid=1072, filesize=11.9 K 2023-04-13 14:54:22,901 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/5cbef882dc52454ba07ab8c7e43466da as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/5cbef882dc52454ba07ab8c7e43466da 2023-04-13 14:54:22,906 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/5cbef882dc52454ba07ab8c7e43466da, entries=150, sequenceid=1072, filesize=11.9 K 2023-04-13 14:54:22,907 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~394.64 KB/404112, currentSize=40.25 KB/41220 for 0b6ed6780bbc11018cb686e788422bc1 in 897ms, sequenceid=1072, compaction requested=true 2023-04-13 14:54:22,907 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:22,907 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:22,907 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:22,907 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:22,907 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,911 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:22,911 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:22,913 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:22,913 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:22,913 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:22,913 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:22,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:22,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:22,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f68bd15[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:22,915 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:22,915 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:22,916 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:22,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:22,917 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:22,917 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:22,917 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:22,917 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,917 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:22,918 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:22,918 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:22,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:22,921 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:22,921 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:22,921 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:22,921 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:22,921 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:22,923 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:22,923 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:22,923 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:22,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:22,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:22,924 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,925 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,928 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:22,928 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:22,928 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:22,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:22,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:22,928 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:22,929 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:22,929 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:22,929 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:22,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:22,928 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:22,929 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:22,929 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:22,929 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:22,929 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:23,029 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:23,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:23,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:23,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:23,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:23,033 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:23,033 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:23,033 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:23,033 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:23,033 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:23,033 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:23,034 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:23,034 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:23,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:23,035 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:23,035 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:23,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:23,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:23,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:23,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:23,036 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:23,036 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:23,036 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:23,036 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:23,036 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:23,036 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:23,036 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:23,036 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 14:54:23,037 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:23,037 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:23,037 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:23,037 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:23,037 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:23,037 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:23,049 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413c6b25c7d165e4e07bcfa58af165a08d2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c6b25c7d165e4e07bcfa58af165a08d2 2023-04-13 14:54:23,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:23,262 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:23,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681397663268, max timestamp=1681397663283Num uniques 859; 2023-04-13 14:54:23,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-13 14:54:23,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397663268, max timestamp=1681397663283Num uniques 859; 2023-04-13 14:54:23,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397663268, max timestamp=1681397663283Num uniques 859; 2023-04-13 14:54:23,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:54:23,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:54:23,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:23,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397723304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:23,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:23,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397723305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:23,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:23,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397723308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:23,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:23,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397723413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:23,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:23,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397723413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:23,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:23,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397723414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:23,455 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1081, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/a0ae1102d4f647269bef0ae7bb8e57d6 2023-04-13 14:54:23,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:23,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397723616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:23,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:23,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397723616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:23,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:23,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397723617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:23,868 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=1081 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/8d8707dafbbf48a4836cd937be434b49 2023-04-13 14:54:23,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:23,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397723921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:23,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:23,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397723922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:23,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:23,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397723923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,281 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=1081 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/e91d9f575ac34da193f21aaec1f9e632 2023-04-13 14:54:24,289 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/a0ae1102d4f647269bef0ae7bb8e57d6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/a0ae1102d4f647269bef0ae7bb8e57d6 2023-04-13 14:54:24,294 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/a0ae1102d4f647269bef0ae7bb8e57d6, entries=150, sequenceid=1081, filesize=25.4 K 2023-04-13 14:54:24,296 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/8d8707dafbbf48a4836cd937be434b49 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/8d8707dafbbf48a4836cd937be434b49 2023-04-13 14:54:24,301 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/8d8707dafbbf48a4836cd937be434b49, entries=150, sequenceid=1081, filesize=11.9 K 2023-04-13 14:54:24,302 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/e91d9f575ac34da193f21aaec1f9e632 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/e91d9f575ac34da193f21aaec1f9e632 2023-04-13 14:54:24,307 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/e91d9f575ac34da193f21aaec1f9e632, entries=150, sequenceid=1081, filesize=11.9 K 2023-04-13 14:54:24,308 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=143.13 KB/146560 for 0b6ed6780bbc11018cb686e788422bc1 in 1272ms, sequenceid=1081, compaction requested=true 2023-04-13 14:54:24,308 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:24,308 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:24,308 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:24,308 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:24,308 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,308 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:24,308 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:24,310 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,310 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,310 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,310 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:24,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f4db533[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:24,312 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:24,312 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,313 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,313 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:24,313 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,313 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:24,313 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:24,313 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:24,322 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:24,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:24,326 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:24,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:24,328 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:24,328 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:24,328 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:24,328 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:24,328 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:24,332 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,332 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,332 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:24,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:24,333 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,334 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:24,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:24,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:24,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:24,338 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:24,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:24,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:24,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:24,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:24,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:24,338 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:24,339 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:24,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:24,433 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=149.83 KB heapSize=425.92 KB 2023-04-13 14:54:24,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7680, totalHeapSize=17712, min timestamp=1681397663303, max timestamp=1681397664431Num uniques 1218; 2023-04-13 14:54:24,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=55770, new segment data size=7680, suffix heap size=154640,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1218, new segment cells count=168 2023-04-13 14:54:24,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397663303, max timestamp=1681397664431Num uniques 1018; 2023-04-13 14:54:24,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 14:54:24,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397663303, max timestamp=1681397664431Num uniques 1018; 2023-04-13 14:54:24,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-13 14:54:24,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:24,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:24,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:24,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:24,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:24,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:24,439 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:24,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:24,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:24,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:24,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:24,446 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:24,446 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:24,447 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:24,447 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:24,447 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,447 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,448 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:24,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:24,448 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,450 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,450 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:24,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:24,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:24,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:24,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:24,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:24,460 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:24,460 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:24,460 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:24,460 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:24,460 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:24,460 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:24,460 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:24,460 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:24,460 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,460 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:24,460 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:24,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,462 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,463 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,463 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:24,464 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:24,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1503e830[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:24,464 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,466 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,467 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:24,467 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:24,467 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:24,467 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,467 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:24,468 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:24,471 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:24,471 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:24,471 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:24,471 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:24,471 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:24,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,476 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:24,476 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:24,476 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,477 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397664465, max timestamp=1681397664479Num uniques 859; 2023-04-13 14:54:24,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681397664451, max timestamp=1681397664478Num uniques 859; 2023-04-13 14:54:24,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:54:24,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-13 14:54:24,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397664465, max timestamp=1681397664479Num uniques 859; 2023-04-13 14:54:24,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,487 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:24,487 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:24,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:24,488 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,488 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,488 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,488 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,488 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:24,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:24,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:24,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:24,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:24,488 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,488 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,488 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:24,488 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:24,488 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:24,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:24,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397724510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:24,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397724515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1681397664500, max timestamp=1681397664515Num uniques 1018; 2023-04-13 14:54:24,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-04-13 14:54:24,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1681397664500, max timestamp=1681397664515Num uniques 1218; 2023-04-13 14:54:24,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=55778, new segment data size=16848, suffix heap size=154640,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1218, new segment cells count=368 2023-04-13 14:54:24,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1681397664500, max timestamp=1681397664515Num uniques 1018; 2023-04-13 14:54:24,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-04-13 14:54:24,588 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:24,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:24,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:24,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:24,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:24,593 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:24,593 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:24,593 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:24,593 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:24,593 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,593 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:24,594 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:24,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:24,595 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,595 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:24,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:24,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:24,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:24,596 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:24,596 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:24,596 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:24,596 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:24,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:24,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397724593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,598 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,598 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,598 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,598 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:24,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b679b54[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:24,600 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:24,600 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,601 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,601 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:24,601 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:24,601 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:24,601 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,601 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:24,603 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,603 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,603 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:24,603 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:24,606 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:24,606 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:24,606 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:24,606 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:24,606 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:24,608 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,608 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,608 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:24,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,610 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:24,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:24,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:24,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,613 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,613 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:24,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:24,613 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:24,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:24,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:24,614 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:24,614 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:24,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:24,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397724618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:24,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397724619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:24,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397724698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,714 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:24,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:24,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:24,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:24,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:24,717 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:24,717 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:24,718 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:24,718 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:24,718 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,718 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,719 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,719 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:24,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:24,719 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,719 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:24,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:24,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:24,720 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:24,721 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:24,721 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:24,721 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:24,721 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:24,723 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,723 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,724 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,724 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:24,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@716d1138[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:24,726 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:24,726 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,727 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:24,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:24,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:24,727 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,727 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:24,728 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:24,728 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:24,728 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,728 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:24,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:24,731 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:24,731 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:24,731 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:24,731 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:24,731 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:24,733 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,733 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,733 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:24,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:24,733 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,735 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,739 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:24,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:24,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:24,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,739 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:24,739 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:24,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:24,740 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:24,741 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:24,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:24,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:24,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:24,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:24,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:24,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397724825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:24,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397724825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,839 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:24,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:24,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:24,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:24,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:24,843 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:24,843 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:24,843 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:24,843 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:24,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,845 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:24,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:24,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:24,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:24,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:24,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:24,846 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:24,846 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:24,846 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:24,846 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:24,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:24,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:24,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:24,848 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,848 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,848 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,848 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:24,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37bd60b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:24,850 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:24,850 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,851 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:24,851 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:24,851 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:24,851 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,851 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:24,854 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,854 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,854 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:24,854 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:24,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:24,857 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:24,857 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:24,857 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:24,857 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:24,857 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:24,858 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413a0536f5971b84ec4a34658f188698b10 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413a0536f5971b84ec4a34658f188698b10 2023-04-13 14:54:24,859 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,859 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:24,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,859 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,860 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,862 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,864 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:24,865 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:24,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:24,865 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:24,865 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:24,865 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:24,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:24,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:24,865 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,865 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,865 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,865 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,865 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,865 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:24,865 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:24,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:24,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:24,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397724901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,966 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:24,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:24,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:24,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:24,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:24,969 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:24,969 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:24,969 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:24,969 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:24,970 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,970 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,972 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:24,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:24,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:24,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:24,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:24,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:24,973 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:24,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:24,974 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:24,974 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:24,974 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:24,974 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:24,976 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,976 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,976 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,976 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:24,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b3ec3c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:24,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:24,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,979 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:24,979 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,979 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:24,979 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:24,979 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:24,980 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:24,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:24,983 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:24,983 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:24,983 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:24,983 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:24,983 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:24,986 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,986 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:24,986 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:24,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:24,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:24,986 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,988 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,989 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:24,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:24,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:24,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:24,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:24,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:24,990 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:24,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:24,990 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:24,990 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:24,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:24,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:24,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:25,090 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:25,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:25,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:25,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:25,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:25,094 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:25,094 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,094 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:25,094 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:25,095 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,095 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,096 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:25,096 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:25,096 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,096 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:25,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:25,101 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:25,101 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,101 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:25,102 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:25,102 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:25,102 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,102 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:25,102 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:25,102 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,102 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:25,102 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:25,102 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,102 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:25,102 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:25,104 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,104 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,104 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,104 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ec96b65[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:25,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:25,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,108 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:25,108 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,108 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:25,109 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:25,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:25,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:25,112 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:25,112 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:25,112 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:25,112 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:25,112 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:25,113 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,113 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,113 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,114 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:25,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:25,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:25,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:25,117 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:25,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:25,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:25,118 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:25,118 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:25,118 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,118 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:25,118 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,118 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:25,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397725130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:25,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397725130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:25,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397725208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,218 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:25,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:25,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:25,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:25,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:25,222 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:25,222 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,222 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:25,222 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:25,222 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,222 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,227 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:25,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:25,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:25,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:25,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:25,229 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,229 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:25,229 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:25,229 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:25,231 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,231 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,231 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,231 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:25,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41122 deadline: 1681397725230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:25,234 DEBUG [Thread-843] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:54:25,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57344755[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:25,234 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:25,234 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,235 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,235 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:25,236 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,236 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:25,237 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,237 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:25,237 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:25,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41140 deadline: 1681397725235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,237 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,239 DEBUG [Thread-847] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:25,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:25,241 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:25,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:25,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:25,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:25,241 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:25,243 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,243 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:25,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,243 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:25,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,245 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,251 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:25,251 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,251 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:25,251 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,252 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:25,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:25,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:25,252 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,252 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,252 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:25,252 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:25,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:25,252 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:25,252 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:25,252 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:25,266 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1121, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/7e30f17143b348e98d3380d72824da1a 2023-04-13 14:54:25,352 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:25,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:25,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:25,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:25,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:25,355 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:25,355 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,355 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:25,355 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:25,356 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,356 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,357 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,357 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:25,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:25,357 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,357 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:25,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:25,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:25,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:25,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:25,359 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,359 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:25,359 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:25,359 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:25,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:25,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:25,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:25,362 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,362 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,362 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,362 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:25,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa21836[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:25,364 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:25,364 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,365 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,365 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:25,365 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:25,365 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,365 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:25,365 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:25,366 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:25,366 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,366 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,367 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:25,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:25,369 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:25,369 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:25,369 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:25,369 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:25,370 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:25,371 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,371 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,371 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:25,372 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,379 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,382 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:25,382 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:25,382 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:25,382 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:25,382 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:25,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:25,383 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:25,383 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,383 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,383 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,383 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,383 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:25,383 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:25,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:25,483 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:25,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:25,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:25,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:25,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:25,486 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:25,486 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,486 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:25,486 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:25,487 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,487 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,488 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:25,488 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:25,488 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,488 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:25,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:25,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:25,490 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:25,490 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:25,490 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,490 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:25,490 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:25,490 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,490 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:25,490 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:25,490 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,490 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:25,490 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:25,491 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,491 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,492 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,492 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:25,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce6d752[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:25,493 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:25,493 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,494 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,495 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:25,495 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:25,495 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:25,495 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,495 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:25,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,497 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:25,497 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,497 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:25,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:25,499 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:25,499 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:25,499 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:25,499 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:25,499 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:25,501 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,501 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,501 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:25,501 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,502 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,504 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:25,504 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:25,505 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:25,504 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:25,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:25,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:25,504 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,505 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,505 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,505 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,505 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,505 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:25,505 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:25,505 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:25,605 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:25,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:25,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:25,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:25,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:25,608 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:25,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:25,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:25,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,610 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,610 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:25,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:25,611 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,611 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:25,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:25,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:25,612 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,613 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:25,613 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:25,613 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,613 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:25,613 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:25,613 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,613 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:25,613 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:25,615 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,615 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,615 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,615 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:25,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19f105e8[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:25,617 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:25,617 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,618 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,618 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,618 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:25,618 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:25,618 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:25,618 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:25,619 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,619 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:25,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,619 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:25,620 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:25,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:25,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:25,622 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:25,622 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:25,622 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:25,622 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:25,622 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:25,623 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,623 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,623 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:25,624 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,625 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:25,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:25,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:25,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:25,628 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:25,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:25,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:25,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:25,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,628 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:25,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:25,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:25,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,628 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:25,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:25,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397725634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:25,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397725634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,682 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=1121 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/9c8a054c2acd464091af02d4e71201d5 2023-04-13 14:54:25,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:25,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397725714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,728 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:25,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:25,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:25,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:25,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:25,732 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:25,732 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,732 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:25,732 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:25,732 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,732 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,734 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,734 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,734 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:25,734 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:25,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:25,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:25,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:25,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:25,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:25,735 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:25,735 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,735 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:25,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:25,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:25,736 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,736 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:25,736 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:25,736 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:25,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:25,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:25,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:25,745 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,745 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,745 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,745 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:25,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c8af76c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:25,747 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:25,747 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,749 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,749 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:25,749 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:25,749 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,749 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:25,749 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:25,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:25,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:25,750 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,751 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:25,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:25,753 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:25,753 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:25,753 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:25,753 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:25,753 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:25,755 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,755 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,755 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:25,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:25,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:25,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,755 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,757 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:25,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:25,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:25,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:25,759 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:25,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:25,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:25,759 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:25,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:25,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,759 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:25,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:25,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:25,860 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:25,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:25,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:25,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:25,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:25,863 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:25,863 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,863 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:25,863 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:25,863 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,863 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,866 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:25,866 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:25,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:25,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,866 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,867 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:25,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:25,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:25,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:25,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:25,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:25,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:25,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:25,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:25,868 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,868 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:25,868 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:25,868 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,868 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:25,868 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:25,868 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,868 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:25,868 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:25,869 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,869 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,869 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,869 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:25,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76732163[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:25,871 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:25,871 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,872 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:25,872 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:25,872 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:25,872 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,872 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:25,873 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:25,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:25,876 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:25,876 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:25,876 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:25,876 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:25,876 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:25,879 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,879 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,880 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:25,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,880 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:25,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:25,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,881 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,883 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,883 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,883 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,883 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:25,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:25,883 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:25,883 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:25,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:25,883 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,883 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,883 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:25,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:25,883 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:25,883 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,884 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:25,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:25,984 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:25,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:25,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:25,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:25,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:25,987 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:25,987 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,987 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:25,987 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:25,988 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,988 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,989 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:25,989 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,989 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,989 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:25,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:25,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:25,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:25,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:25,990 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:25,990 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:25,991 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,991 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:25,991 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:25,991 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:25,992 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,992 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,993 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,993 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:25,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0534b8[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:25,994 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:25,994 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,996 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:25,997 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,997 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:25,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:25,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:25,998 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:26,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:26,000 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:26,000 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:26,000 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:26,000 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:26,000 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:26,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:26,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:26,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:26,003 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,004 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,006 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,006 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,006 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,006 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:26,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:26,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:26,006 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,006 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,006 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:26,006 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:26,006 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:26,006 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:26,006 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:26,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:26,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:26,094 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=12.62 KB at sequenceid=1121 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/607f9d6122da4459b8179645ba84bb68 2023-04-13 14:54:26,099 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/7e30f17143b348e98d3380d72824da1a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/7e30f17143b348e98d3380d72824da1a 2023-04-13 14:54:26,103 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/7e30f17143b348e98d3380d72824da1a, entries=150, sequenceid=1121, filesize=25.4 K 2023-04-13 14:54:26,104 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/9c8a054c2acd464091af02d4e71201d5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/9c8a054c2acd464091af02d4e71201d5 2023-04-13 14:54:26,107 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:26,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:26,109 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/9c8a054c2acd464091af02d4e71201d5, entries=150, sequenceid=1121, filesize=11.9 K 2023-04-13 14:54:26,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:26,109 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/607f9d6122da4459b8179645ba84bb68 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/607f9d6122da4459b8179645ba84bb68 2023-04-13 14:54:26,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:26,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:26,110 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:26,110 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:26,110 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:26,110 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:26,110 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,111 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,113 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:26,113 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:26,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:26,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,113 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,113 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:26,114 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/607f9d6122da4459b8179645ba84bb68, entries=150, sequenceid=1121, filesize=11.9 K 2023-04-13 14:54:26,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,115 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~39.46 KB/40402, heapSize ~103.36 KB/105840, currentSize=154.31 KB/158010 for 0b6ed6780bbc11018cb686e788422bc1 in 1682ms, sequenceid=1121, compaction requested=true 2023-04-13 14:54:26,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:26,115 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:26,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:26,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:26,115 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 14:54:26,115 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 14:54:26,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:26,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:26,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:26,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:26,116 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:26,116 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:26,116 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:26,116 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:26,116 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=154.31 KB heapSize=425.16 KB 2023-04-13 14:54:26,116 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:26,116 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:26,117 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:26,117 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:26,117 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:26,117 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:26,118 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 156379 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 14:54:26,118 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0b6ed6780bbc11018cb686e788422bc1/A is initiating minor compaction (all files) 2023-04-13 14:54:26,118 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b6ed6780bbc11018cb686e788422bc1/A in TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:26,118 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73219 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 14:54:26,118 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0b6ed6780bbc11018cb686e788422bc1/B is initiating minor compaction (all files) 2023-04-13 14:54:26,118 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b6ed6780bbc11018cb686e788422bc1/B in TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:26,119 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/7882b81f765642bfb6e2789220ad01da, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/d070d8b6feaa4058a22841ad42a9b8bc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/2cc23bc72cfd4350b9267919d17b6733, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/a8c58d644e0e4b599f3b03f6abb1ac32, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/8d8707dafbbf48a4836cd937be434b49, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/9c8a054c2acd464091af02d4e71201d5] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp, totalSize=71.5 K 2023-04-13 14:54:26,118 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/55c72473efd949b6955de42567a861ce, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/88f993cdb8bf41e18685fb8d51663b27, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/b25a91f98dac4d9c8a0292b5f01d5360, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/455ff6ca5b414298966ff23a38f44375, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/a0ae1102d4f647269bef0ae7bb8e57d6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/7e30f17143b348e98d3380d72824da1a] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp, totalSize=152.7 K 2023-04-13 14:54:26,119 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 7882b81f765642bfb6e2789220ad01da, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=970, earliestPutTs=1681397658065 2023-04-13 14:54:26,119 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 55c72473efd949b6955de42567a861ce, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=970, earliestPutTs=1681397658065 2023-04-13 14:54:26,119 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting d070d8b6feaa4058a22841ad42a9b8bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1007, earliestPutTs=1681397658139 2023-04-13 14:54:26,120 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 88f993cdb8bf41e18685fb8d51663b27, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1007, earliestPutTs=1681397658139 2023-04-13 14:54:26,120 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 2cc23bc72cfd4350b9267919d17b6733, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1019, earliestPutTs=1681397658802 2023-04-13 14:54:26,121 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting b25a91f98dac4d9c8a0292b5f01d5360, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1019, earliestPutTs=1681397658802 2023-04-13 14:54:26,121 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting a8c58d644e0e4b599f3b03f6abb1ac32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1072, earliestPutTs=1681397661018 2023-04-13 14:54:26,121 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 455ff6ca5b414298966ff23a38f44375, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1072, earliestPutTs=1681397661018 2023-04-13 14:54:26,121 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 8d8707dafbbf48a4836cd937be434b49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1081, earliestPutTs=1681397662145 2023-04-13 14:54:26,122 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting a0ae1102d4f647269bef0ae7bb8e57d6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1081, earliestPutTs=1681397662145 2023-04-13 14:54:26,122 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 9c8a054c2acd464091af02d4e71201d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1121, earliestPutTs=1681397663307 2023-04-13 14:54:26,122 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 7e30f17143b348e98d3380d72824da1a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1121, earliestPutTs=1681397663307 2023-04-13 14:54:26,135 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041337a87df58ad94a0281839e62be9a8905 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041337a87df58ad94a0281839e62be9a8905 2023-04-13 14:54:26,139 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b6ed6780bbc11018cb686e788422bc1#B#compaction#138 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:26,162 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b6ed6780bbc11018cb686e788422bc1#A#compaction#139 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:26,171 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/e1ab5c327fdf45048f128ebe390772e7 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/e1ab5c327fdf45048f128ebe390772e7 2023-04-13 14:54:26,179 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 0b6ed6780bbc11018cb686e788422bc1/B of 0b6ed6780bbc11018cb686e788422bc1 into e1ab5c327fdf45048f128ebe390772e7(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:26,179 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:26,179 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., storeName=0b6ed6780bbc11018cb686e788422bc1/B, priority=10, startTime=1681397666115; duration=0sec 2023-04-13 14:54:26,180 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:26,180 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 14:54:26,183 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73219 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 14:54:26,183 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0b6ed6780bbc11018cb686e788422bc1/C is initiating minor compaction (all files) 2023-04-13 14:54:26,183 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b6ed6780bbc11018cb686e788422bc1/C in TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:26,183 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/32066e63404641b78e2341df8602a263, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/911cea0cca364293b05af43812b92ab2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/d4d2acd59cf24228925d7ba9a42d3ea1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/5cbef882dc52454ba07ab8c7e43466da, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/e91d9f575ac34da193f21aaec1f9e632, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/607f9d6122da4459b8179645ba84bb68] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp, totalSize=71.5 K 2023-04-13 14:54:26,184 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 32066e63404641b78e2341df8602a263, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=970, earliestPutTs=1681397658065 2023-04-13 14:54:26,184 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 911cea0cca364293b05af43812b92ab2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1007, earliestPutTs=1681397658139 2023-04-13 14:54:26,185 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting d4d2acd59cf24228925d7ba9a42d3ea1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1019, earliestPutTs=1681397658802 2023-04-13 14:54:26,185 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 5cbef882dc52454ba07ab8c7e43466da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1072, earliestPutTs=1681397661018 2023-04-13 14:54:26,186 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting e91d9f575ac34da193f21aaec1f9e632, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1081, earliestPutTs=1681397662145 2023-04-13 14:54:26,187 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 607f9d6122da4459b8179645ba84bb68, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1121, earliestPutTs=1681397663307 2023-04-13 14:54:26,202 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b6ed6780bbc11018cb686e788422bc1#C#compaction#140 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:26,219 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/fa54aab2eea8475f848086c010e00807 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/fa54aab2eea8475f848086c010e00807 2023-04-13 14:54:26,226 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 0b6ed6780bbc11018cb686e788422bc1/C of 0b6ed6780bbc11018cb686e788422bc1 into fa54aab2eea8475f848086c010e00807(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:26,226 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:26,226 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., storeName=0b6ed6780bbc11018cb686e788422bc1/C, priority=10, startTime=1681397666115; duration=0sec 2023-04-13 14:54:26,226 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:26,554 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1174, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/449f73381b3449b496b6ec72a611289f 2023-04-13 14:54:26,571 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=1174 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/5d1c10d06d894e148d320ad736704bef 2023-04-13 14:54:26,594 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=1174 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/0bc6c1da9d2644cdbfc9424a03665591 2023-04-13 14:54:26,598 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/449f73381b3449b496b6ec72a611289f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/449f73381b3449b496b6ec72a611289f 2023-04-13 14:54:26,602 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/449f73381b3449b496b6ec72a611289f, entries=150, sequenceid=1174, filesize=25.4 K 2023-04-13 14:54:26,603 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/5d1c10d06d894e148d320ad736704bef as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/5d1c10d06d894e148d320ad736704bef 2023-04-13 14:54:26,608 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/5d1c10d06d894e148d320ad736704bef, entries=150, sequenceid=1174, filesize=11.9 K 2023-04-13 14:54:26,609 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/0bc6c1da9d2644cdbfc9424a03665591 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0bc6c1da9d2644cdbfc9424a03665591 2023-04-13 14:54:26,617 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0bc6c1da9d2644cdbfc9424a03665591, entries=150, sequenceid=1174, filesize=11.9 K 2023-04-13 14:54:26,618 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~425.11 KB/435312, currentSize=0 B/0 for 0b6ed6780bbc11018cb686e788422bc1 in 502ms, sequenceid=1174, compaction requested=false 2023-04-13 14:54:26,618 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:26,618 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:26,618 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:26,619 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:26,619 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:26,621 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/687bc298888341ff8231cc0d6e3a41f4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/687bc298888341ff8231cc0d6e3a41f4 2023-04-13 14:54:26,622 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,622 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:26,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dcaa495[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:26,625 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:26,625 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,627 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 0b6ed6780bbc11018cb686e788422bc1/A of 0b6ed6780bbc11018cb686e788422bc1 into 687bc298888341ff8231cc0d6e3a41f4(size=25.7 K), total size for store is 51.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:26,627 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,627 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:26,627 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., storeName=0b6ed6780bbc11018cb686e788422bc1/A, priority=10, startTime=1681397666115; duration=0sec 2023-04-13 14:54:26,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,627 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:26,627 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,627 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:26,627 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:26,627 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:26,627 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:26,631 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,631 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:26,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,632 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:26,632 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:26,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:26,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:26,635 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:26,635 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:26,635 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:26,635 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:26,635 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:26,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:26,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:26,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:26,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:26,638 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:26,640 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:26,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:26,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:26,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:26,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:26,643 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:26,643 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:26,643 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:26,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,644 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:26,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:26,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:26,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:26,672 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:54:26,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:26,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:26,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:26,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:26,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:26,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:26,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:26,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397726719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681397666677, max timestamp=1681397666721Num uniques 859; 2023-04-13 14:54:26,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681397666677, max timestamp=1681397666721Num uniques 859; 2023-04-13 14:54:26,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-13 14:54:26,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681397666677, max timestamp=1681397666721Num uniques 859; 2023-04-13 14:54:26,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-13 14:54:26,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-13 14:54:26,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:26,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397726727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:26,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397726730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,743 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:26,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:26,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:26,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:26,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:26,746 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:26,746 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:26,747 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:26,747 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:26,747 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,747 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,748 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:26,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:26,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:26,749 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,749 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:26,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:26,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:26,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:26,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:26,750 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:26,750 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:26,750 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:26,750 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:26,753 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:26,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76400bd7[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:26,754 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:26,754 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,756 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,756 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:26,756 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:26,756 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:26,756 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,756 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:26,757 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,757 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:26,757 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:26,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:26,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:26,757 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:26,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:26,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:26,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:26,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:26,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:26,759 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:26,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:26,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:26,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:26,760 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:26,761 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,761 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:26,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:26,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,761 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:26,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:26,762 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:26,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,764 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:26,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,767 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:26,767 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:26,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:26,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:26,767 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:26,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:26,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:26,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:26,768 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,768 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,768 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:26,768 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,768 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:26,768 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,768 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,768 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:26,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:26,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397726824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:26,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397726831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:26,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397726833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,868 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:26,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:26,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:26,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:26,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:26,871 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:26,871 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:26,871 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:26,871 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:26,872 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,872 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,873 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:26,873 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:26,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:26,873 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,873 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:26,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:26,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:26,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:26,874 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:26,874 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:26,874 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:26,875 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:26,875 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:26,875 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:26,875 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:26,875 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:26,875 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:26,875 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:26,875 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:26,875 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,875 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:26,875 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:26,876 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,876 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:26,877 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,877 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:26,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:26,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19d75445[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:26,878 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:26,878 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,879 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,879 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:26,880 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:26,880 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:26,880 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,880 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:26,882 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,882 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:26,882 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:26,882 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:26,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:26,884 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:26,884 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:26,884 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:26,884 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:26,885 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:26,886 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,886 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:26,887 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:26,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,886 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,888 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,892 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,892 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,892 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,892 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,892 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:26,892 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:26,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:26,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:26,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:26,892 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,893 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:26,893 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:26,893 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:26,893 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:26,893 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:26,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:26,993 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:26,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:26,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:26,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:26,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:26,997 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:26,997 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:26,997 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:26,997 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:26,997 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,997 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,998 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:26,998 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:26,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:26,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:26,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:27,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:27,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:27,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:27,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:27,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:27,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:27,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:27,000 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:27,000 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:27,000 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:27,000 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:27,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:27,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:27,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:27,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:27,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,002 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:27,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:27,004 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:27,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c020333[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:27,004 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,005 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,005 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:27,005 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:27,005 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,005 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:27,005 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:27,007 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,007 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:27,007 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:27,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,007 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:27,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:27,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:27,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:27,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:27,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:27,009 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:27,009 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:27,009 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:27,009 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:27,009 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:27,015 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,015 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:27,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,015 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:27,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:27,015 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,016 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,019 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:27,019 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:27,019 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,019 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:27,019 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:27,019 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:27,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:27,019 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,019 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,019 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,019 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,019 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:27,019 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:27,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:27,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:27,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:27,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397727028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:27,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397727033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:27,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397727036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,100 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041382b24851b39f4c4ea06c52fbfea5630b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041382b24851b39f4c4ea06c52fbfea5630b 2023-04-13 14:54:27,119 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:27,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:27,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:27,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:27,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:27,124 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:27,124 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:27,124 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:27,124 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:27,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,126 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:27,126 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:27,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:27,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,127 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,127 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:27,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:27,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:27,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:27,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:27,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:27,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:27,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:27,133 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:27,133 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:27,133 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:27,133 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:27,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:27,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:27,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:27,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:27,135 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:27,135 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,135 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,136 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:27,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:27,137 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:27,137 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f4ec2a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:27,140 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,140 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:27,140 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,140 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:27,140 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:27,140 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:27,142 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:27,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:27,144 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:27,144 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:27,144 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:27,144 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:27,144 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:27,146 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,146 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:27,146 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,147 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,148 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:27,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:27,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,151 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,151 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,151 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:27,151 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:27,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:27,151 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,151 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:27,151 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:27,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:27,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:27,251 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:27,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:27,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:27,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:27,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:27,255 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:27,255 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:27,255 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:27,255 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:27,255 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,255 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,257 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:27,257 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:27,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:27,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:27,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:27,259 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:27,259 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:27,259 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:27,259 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:27,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:27,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,261 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:27,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:27,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:27,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21c4fd0[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:27,263 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:27,263 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,264 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:27,264 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:27,264 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,264 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:27,264 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:27,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:27,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:27,268 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:27,268 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:27,268 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:27,268 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:27,268 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:27,271 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,271 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:27,271 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:27,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:27,272 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,273 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,276 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:27,276 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:27,277 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:27,277 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:27,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:27,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:27,277 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,277 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:27,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:27,277 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,277 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:27,277 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:27,277 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,277 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,277 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:27,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397727334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:27,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397727334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:27,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397727339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,377 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:27,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:27,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:27,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:27,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:27,381 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:27,381 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:27,381 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:27,381 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:27,381 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,381 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,382 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,382 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:27,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:27,383 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,383 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:27,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:27,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:27,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:27,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:27,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:27,385 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:27,385 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:27,385 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:27,385 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:27,386 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,386 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:27,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:27,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:27,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21ca56aa[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:27,388 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:27,388 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:27,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:27,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:27,391 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,391 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:27,392 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:27,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:27,395 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:27,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:27,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:27,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:27,395 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:27,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:27,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:27,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:27,397 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,398 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:27,404 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:27,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:27,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:27,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:27,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,404 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:27,404 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:27,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:27,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:27,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:27,504 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:27,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:27,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:27,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:27,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:27,507 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:27,507 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:27,507 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:27,507 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:27,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,509 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:27,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:27,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:27,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:27,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:27,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:27,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:27,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:27,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:27,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:27,511 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:27,511 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:27,511 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:27,511 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:27,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:27,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:27,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:27,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:27,512 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,512 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:27,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,514 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1188, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/0792d828a74244a39b18d6cad9a3536f 2023-04-13 14:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:27,514 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dadd23b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:27,515 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,516 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:27,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:27,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:27,516 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,516 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:27,522 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,525 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1188 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/58e99f0c12c04c6ba437585f75c41bbb 2023-04-13 14:54:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:27,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:27,525 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:27,525 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:27,526 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:27,526 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:27,526 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:27,528 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,528 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,528 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:27,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:27,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,530 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:27,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:27,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:27,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:27,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:27,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:27,533 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:27,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:27,533 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,533 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:27,533 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:27,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:27,533 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:27,533 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1188 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/8014d90b9a27451aa6aff695f2a0fab8 2023-04-13 14:54:27,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/0792d828a74244a39b18d6cad9a3536f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/0792d828a74244a39b18d6cad9a3536f 2023-04-13 14:54:27,545 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/0792d828a74244a39b18d6cad9a3536f, entries=400, sequenceid=1188, filesize=59.8 K 2023-04-13 14:54:27,545 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/58e99f0c12c04c6ba437585f75c41bbb as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/58e99f0c12c04c6ba437585f75c41bbb 2023-04-13 14:54:27,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/58e99f0c12c04c6ba437585f75c41bbb, entries=150, sequenceid=1188, filesize=11.9 K 2023-04-13 14:54:27,550 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/8014d90b9a27451aa6aff695f2a0fab8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/8014d90b9a27451aa6aff695f2a0fab8 2023-04-13 14:54:27,557 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/8014d90b9a27451aa6aff695f2a0fab8, entries=150, sequenceid=1188, filesize=11.9 K 2023-04-13 14:54:27,558 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=147.60 KB/151140 for 0b6ed6780bbc11018cb686e788422bc1 in 886ms, sequenceid=1188, compaction requested=true 2023-04-13 14:54:27,558 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:27,558 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:27,558 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:27,558 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:27,558 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:27,558 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 14:54:27,559 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 113654 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:27,559 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0b6ed6780bbc11018cb686e788422bc1/A is initiating minor compaction (all files) 2023-04-13 14:54:27,560 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b6ed6780bbc11018cb686e788422bc1/A in TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:27,560 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:27,560 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/687bc298888341ff8231cc0d6e3a41f4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/449f73381b3449b496b6ec72a611289f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/0792d828a74244a39b18d6cad9a3536f] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp, totalSize=111.0 K 2023-04-13 14:54:27,560 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0b6ed6780bbc11018cb686e788422bc1/B is initiating minor compaction (all files) 2023-04-13 14:54:27,560 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b6ed6780bbc11018cb686e788422bc1/B in TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:27,560 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/e1ab5c327fdf45048f128ebe390772e7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/5d1c10d06d894e148d320ad736704bef, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/58e99f0c12c04c6ba437585f75c41bbb] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp, totalSize=36.0 K 2023-04-13 14:54:27,561 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 687bc298888341ff8231cc0d6e3a41f4, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=1121, earliestPutTs=1681397663307 2023-04-13 14:54:27,561 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting e1ab5c327fdf45048f128ebe390772e7, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=1121, earliestPutTs=1681397663307 2023-04-13 14:54:27,561 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 5d1c10d06d894e148d320ad736704bef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1174, earliestPutTs=1681397664515 2023-04-13 14:54:27,562 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 449f73381b3449b496b6ec72a611289f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1174, earliestPutTs=1681397664515 2023-04-13 14:54:27,562 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 58e99f0c12c04c6ba437585f75c41bbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1188, earliestPutTs=1681397666662 2023-04-13 14:54:27,562 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 0792d828a74244a39b18d6cad9a3536f, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=1188, earliestPutTs=1681397666651 2023-04-13 14:54:27,578 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b6ed6780bbc11018cb686e788422bc1#B#compaction#146 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:27,583 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b6ed6780bbc11018cb686e788422bc1#A#compaction#147 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:27,633 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:27,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:27,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:27,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:27,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:27,636 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:27,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:27,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:27,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:27,637 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,637 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:27,638 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:27,638 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,638 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:27,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:27,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:27,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:27,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:27,639 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:27,639 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:27,639 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:27,639 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:27,639 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:27,639 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:27,639 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:27,639 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=147.60 KB heapSize=359.20 KB 2023-04-13 14:54:27,640 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:27,640 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:27,640 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:27,640 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:27,640 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:27,640 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:27,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:27,842 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:27,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:27,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397727852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:27,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397727853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:27,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397727853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:27,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397727954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:27,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397727956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:27,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397727956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:27,995 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/b619b7ed2ab94947842cf1cc8ecb58c8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/b619b7ed2ab94947842cf1cc8ecb58c8 2023-04-13 14:54:28,000 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b6ed6780bbc11018cb686e788422bc1/B of 0b6ed6780bbc11018cb686e788422bc1 into b619b7ed2ab94947842cf1cc8ecb58c8(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:28,000 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:28,000 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., storeName=0b6ed6780bbc11018cb686e788422bc1/B, priority=13, startTime=1681397667558; duration=0sec 2023-04-13 14:54:28,000 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:28,000 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:28,001 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:28,002 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0b6ed6780bbc11018cb686e788422bc1/C is initiating minor compaction (all files) 2023-04-13 14:54:28,002 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b6ed6780bbc11018cb686e788422bc1/C in TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:28,002 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/fa54aab2eea8475f848086c010e00807, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0bc6c1da9d2644cdbfc9424a03665591, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/8014d90b9a27451aa6aff695f2a0fab8] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp, totalSize=36.0 K 2023-04-13 14:54:28,002 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting fa54aab2eea8475f848086c010e00807, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=1121, earliestPutTs=1681397663307 2023-04-13 14:54:28,003 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 0bc6c1da9d2644cdbfc9424a03665591, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1174, earliestPutTs=1681397664515 2023-04-13 14:54:28,003 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 8014d90b9a27451aa6aff695f2a0fab8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1188, earliestPutTs=1681397666662 2023-04-13 14:54:28,012 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b6ed6780bbc11018cb686e788422bc1#C#compaction#149 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:28,016 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/54f8b8735dab426d8aa318291e03cba3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/54f8b8735dab426d8aa318291e03cba3 2023-04-13 14:54:28,020 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b6ed6780bbc11018cb686e788422bc1/A of 0b6ed6780bbc11018cb686e788422bc1 into 54f8b8735dab426d8aa318291e03cba3(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:28,020 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:28,020 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., storeName=0b6ed6780bbc11018cb686e788422bc1/A, priority=13, startTime=1681397667558; duration=0sec 2023-04-13 14:54:28,020 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:28,027 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/1c0f8311662044acaf07b63554399451 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/1c0f8311662044acaf07b63554399451 2023-04-13 14:54:28,033 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b6ed6780bbc11018cb686e788422bc1/C of 0b6ed6780bbc11018cb686e788422bc1 into 1c0f8311662044acaf07b63554399451(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:28,033 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:28,033 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., storeName=0b6ed6780bbc11018cb686e788422bc1/C, priority=13, startTime=1681397667558; duration=0sec 2023-04-13 14:54:28,033 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:28,056 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304133927256efe8b4e17b388931d387e096b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133927256efe8b4e17b388931d387e096b 2023-04-13 14:54:28,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:28,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397728159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:28,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:28,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397728159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:28,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:28,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397728160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:28,464 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1213, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/c988bab369834edcae9483b7c44f8a00 2023-04-13 14:54:28,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:28,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397728462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:28,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:28,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397728463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:28,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:28,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397728464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:28,875 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1213 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/6f8c4cd1af054616a6c73b0914d57fea 2023-04-13 14:54:28,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:28,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397728966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:28,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:28,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397728968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:28,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:28,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397728969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,286 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1213 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/26f935bf27644a9c9271d00399326d79 2023-04-13 14:54:29,291 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/c988bab369834edcae9483b7c44f8a00 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/c988bab369834edcae9483b7c44f8a00 2023-04-13 14:54:29,296 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/c988bab369834edcae9483b7c44f8a00, entries=150, sequenceid=1213, filesize=25.4 K 2023-04-13 14:54:29,298 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/6f8c4cd1af054616a6c73b0914d57fea as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/6f8c4cd1af054616a6c73b0914d57fea 2023-04-13 14:54:29,303 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/6f8c4cd1af054616a6c73b0914d57fea, entries=150, sequenceid=1213, filesize=11.9 K 2023-04-13 14:54:29,304 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/26f935bf27644a9c9271d00399326d79 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/26f935bf27644a9c9271d00399326d79 2023-04-13 14:54:29,308 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/26f935bf27644a9c9271d00399326d79, entries=150, sequenceid=1213, filesize=11.9 K 2023-04-13 14:54:29,309 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~359.16 KB/367776, currentSize=60.38 KB/61830 for 0b6ed6780bbc11018cb686e788422bc1 in 1670ms, sequenceid=1213, compaction requested=false 2023-04-13 14:54:29,309 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:29,309 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:29,312 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:29,312 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:29,312 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:29,312 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:29,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:29,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eebf018[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:29,314 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:29,314 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:29,314 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:29,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:29,315 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:29,315 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,315 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:29,316 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:29,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:29,319 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:29,319 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:29,319 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:29,319 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:29,319 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:29,321 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:29,321 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:29,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:29,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:29,322 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:29,323 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,324 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,327 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:29,327 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:29,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:29,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:29,329 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,329 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:29,329 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:29,329 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,329 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:29,329 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:29,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:29,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:29,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:29,329 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:29,329 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:29,329 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:29,329 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:29,429 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:29,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:29,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:29,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:29,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:29,433 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:29,433 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:29,433 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:29,433 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:29,433 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:29,433 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:29,434 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:29,434 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:29,435 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,435 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:29,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:29,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:29,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:29,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:29,441 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:29,441 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:29,441 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:29,448 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:29,448 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:29,448 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 14:54:29,448 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:29,448 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:29,449 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:29,449 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:29,449 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:29,449 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:29,475 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041356a604766fa2420d9784a436ca222151 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041356a604766fa2420d9784a436ca222151 2023-04-13 14:54:29,480 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1228, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/d41d29abd2da4f6c9b50d5b316ba7f25 2023-04-13 14:54:29,893 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1228 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/931282876f404e948c7e000b1bb10d25 2023-04-13 14:54:29,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:29,976 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:29,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:29,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397729994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:29,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:29,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 426 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397729995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:29,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397729996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:30,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:30,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397730099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 428 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397730100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:30,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397730100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:30,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397730302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:30,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 430 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397730303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:30,305 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1228 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/688570deda46432c9f1be821f881c033 2023-04-13 14:54:30,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397730303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,310 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/d41d29abd2da4f6c9b50d5b316ba7f25 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/d41d29abd2da4f6c9b50d5b316ba7f25 2023-04-13 14:54:30,315 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/d41d29abd2da4f6c9b50d5b316ba7f25, entries=150, sequenceid=1228, filesize=25.4 K 2023-04-13 14:54:30,315 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/931282876f404e948c7e000b1bb10d25 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/931282876f404e948c7e000b1bb10d25 2023-04-13 14:54:30,319 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/931282876f404e948c7e000b1bb10d25, entries=150, sequenceid=1228, filesize=11.9 K 2023-04-13 14:54:30,320 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/688570deda46432c9f1be821f881c033 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/688570deda46432c9f1be821f881c033 2023-04-13 14:54:30,324 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/688570deda46432c9f1be821f881c033, entries=150, sequenceid=1228, filesize=11.9 K 2023-04-13 14:54:30,324 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 0b6ed6780bbc11018cb686e788422bc1 in 876ms, sequenceid=1228, compaction requested=true 2023-04-13 14:54:30,324 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:30,324 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:30,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:30,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:30,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:30,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:30,326 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,326 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:30,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:30,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:30,326 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:30,326 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:30,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:30,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:30,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:30,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:30,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:30,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee94cda[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:30,328 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:30,328 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:30,329 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:30,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:30,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:30,329 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:30,329 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:30,329 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:30,329 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,329 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:30,331 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,331 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:30,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:30,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:30,331 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:30,332 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:30,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:30,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:30,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:30,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:30,334 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:30,334 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:30,334 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:30,334 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:30,334 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:30,335 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:30,336 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:30,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:30,336 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:30,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:30,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:30,338 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,343 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,343 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:30,343 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:30,343 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:30,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:30,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:30,343 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:30,343 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,343 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:30,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:30,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:30,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:30,343 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:30,344 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:30,344 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:30,344 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:30,344 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:30,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:30,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:30,444 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:30,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:30,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:30,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:30,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:30,448 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:30,448 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:30,448 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:30,448 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:30,448 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:30,448 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:30,450 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:30,450 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:30,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:30,450 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,451 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:30,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:30,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:30,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:30,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:30,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:30,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:30,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:30,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:30,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:30,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:30,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:30,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:30,452 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:30,452 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:30,452 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 14:54:30,453 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:30,453 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:30,453 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:30,453 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:30,453 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:30,453 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:30,476 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413d3ada7ec4d5f46cd938cca2db8b8558a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d3ada7ec4d5f46cd938cca2db8b8558a 2023-04-13 14:54:30,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:30,606 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:30,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:30,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 434 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397730613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:30,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397730615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:30,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397730617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:30,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 436 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397730718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:30,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397730718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:30,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397730719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,890 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1248, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/d4f07cf0e84f44058e996683914d3e7c 2023-04-13 14:54:30,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:30,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:30,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397730920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 438 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397730920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:30,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:30,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397730922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:31,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397731224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:31,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 440 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397731225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:31,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397731226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,302 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1248 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/00866c4803604804be9fab74d4150167 2023-04-13 14:54:31,714 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1248 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/7a68818da0ce4e9bbcc2cfd92b15e98e 2023-04-13 14:54:31,719 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/d4f07cf0e84f44058e996683914d3e7c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/d4f07cf0e84f44058e996683914d3e7c 2023-04-13 14:54:31,729 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/d4f07cf0e84f44058e996683914d3e7c, entries=150, sequenceid=1248, filesize=25.4 K 2023-04-13 14:54:31,730 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/00866c4803604804be9fab74d4150167 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/00866c4803604804be9fab74d4150167 2023-04-13 14:54:31,734 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/00866c4803604804be9fab74d4150167, entries=150, sequenceid=1248, filesize=11.9 K 2023-04-13 14:54:31,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:31,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 442 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397731730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:31,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397731730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:31,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397731731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,735 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/7a68818da0ce4e9bbcc2cfd92b15e98e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/7a68818da0ce4e9bbcc2cfd92b15e98e 2023-04-13 14:54:31,739 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/7a68818da0ce4e9bbcc2cfd92b15e98e, entries=150, sequenceid=1248, filesize=11.9 K 2023-04-13 14:54:31,739 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 0b6ed6780bbc11018cb686e788422bc1 in 1287ms, sequenceid=1248, compaction requested=true 2023-04-13 14:54:31,740 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:31,740 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:31,740 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:31,740 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:31,740 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,740 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:31,740 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:31,741 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:31,741 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:31,742 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:31,742 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:31,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:31,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1de528ab[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:31,743 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:31,743 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:31,744 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:31,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:31,744 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:31,744 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:31,744 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:31,744 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,744 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:31,746 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,746 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:31,746 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:31,746 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:31,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:31,748 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:31,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:31,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:31,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:31,748 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:31,750 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:31,750 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:31,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:31,750 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:31,751 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:31,752 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,754 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,754 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:31,754 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:31,754 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:31,754 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:31,755 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:31,754 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:31,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:31,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:31,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:31,754 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:31,755 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:31,755 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:31,755 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:31,755 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:31,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:31,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:31,855 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:31,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:31,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:31,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:31,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:31,858 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:31,858 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:31,859 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:31,859 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:31,859 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:31,859 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:31,860 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:31,860 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:31,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:31,861 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:31,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:31,861 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:31,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:31,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:31,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:31,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:31,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:31,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:31,862 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:31,862 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:31,862 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:31,862 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:31,862 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:31,862 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:31,862 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:31,862 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 14:54:31,863 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:31,863 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:31,863 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:31,863 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:31,863 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:31,863 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:31,876 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304136e6abd2cb6dd421aa27fcb6c36fc90b0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136e6abd2cb6dd421aa27fcb6c36fc90b0 2023-04-13 14:54:31,882 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1260, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/3bdff373e70e4ab69df4458aef07bca3 2023-04-13 14:54:32,293 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1260 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/44a18a1f9c774f5e99c67a63965e8e6c 2023-04-13 14:54:32,705 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1260 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/b446bdcabb244adc81b83b8788a99b3b 2023-04-13 14:54:32,710 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/3bdff373e70e4ab69df4458aef07bca3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/3bdff373e70e4ab69df4458aef07bca3 2023-04-13 14:54:32,714 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/3bdff373e70e4ab69df4458aef07bca3, entries=150, sequenceid=1260, filesize=25.4 K 2023-04-13 14:54:32,715 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/44a18a1f9c774f5e99c67a63965e8e6c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/44a18a1f9c774f5e99c67a63965e8e6c 2023-04-13 14:54:32,719 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/44a18a1f9c774f5e99c67a63965e8e6c, entries=150, sequenceid=1260, filesize=11.9 K 2023-04-13 14:54:32,720 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/b446bdcabb244adc81b83b8788a99b3b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/b446bdcabb244adc81b83b8788a99b3b 2023-04-13 14:54:32,728 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/b446bdcabb244adc81b83b8788a99b3b, entries=150, sequenceid=1260, filesize=11.9 K 2023-04-13 14:54:32,729 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 0b6ed6780bbc11018cb686e788422bc1 in 867ms, sequenceid=1260, compaction requested=true 2023-04-13 14:54:32,729 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:32,729 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:32,729 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:32,729 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:32,729 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,729 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:32,729 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:32,731 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,731 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:32,731 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,731 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:32,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:32,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:32,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eea0d26[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:32,733 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:32,733 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,735 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:32,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:32,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:32,736 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,736 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:32,737 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:32,737 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:32,737 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:32,738 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:32,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:32,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:32,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:32,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:32,741 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:32,741 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:32,741 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:32,741 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:32,741 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:32,743 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,743 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:32,743 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:32,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:32,744 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,745 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,748 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,748 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:32,748 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:32,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:32,748 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:32,748 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,748 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:32,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:32,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:32,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:32,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:32,748 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,748 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,748 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,748 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:32,748 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:32,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:32,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:32,806 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:54:32,807 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:32,807 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:32,807 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:32,807 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:32,807 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:32,807 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:32,849 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:32,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397672841, max timestamp=1681397672847Num uniques 859; 2023-04-13 14:54:32,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:32,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:54:32,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397672841, max timestamp=1681397672847Num uniques 859; 2023-04-13 14:54:32,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397672841, max timestamp=1681397672847Num uniques 859; 2023-04-13 14:54:32,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:54:32,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:54:32,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:32,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:32,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:32,852 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:32,852 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:32,852 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:32,852 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:32,852 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,852 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:32,854 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,854 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:32,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:32,855 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,855 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:32,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:32,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:32,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:32,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:32,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:32,856 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:32,856 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:32,856 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:32,856 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:32,856 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:32,856 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:32,856 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:32,857 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:32,857 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:32,857 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:32,857 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:32,857 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,857 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:32,857 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:32,858 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,858 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:32,859 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,859 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:32,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:32,860 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:32,860 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f5043d3[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:32,861 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,861 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:32,861 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:32,861 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:32,861 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,861 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:32,862 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,862 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:32,863 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:32,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:32,863 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:32,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:32,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:32,865 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:32,865 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:32,865 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:32,865 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:32,865 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:32,868 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,868 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,868 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:32,868 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:32,869 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:32,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:32,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:32,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:32,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:32,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:32,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,872 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:32,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:32,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:32,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:32,872 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:32,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,872 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:32,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:32,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:32,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 457 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397732870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:32,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397732870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:32,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397732871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,972 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:32,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:32,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:32,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:32,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:32,976 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:32,976 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:32,976 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:32,976 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:32,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:32,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:32,978 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:32,978 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,978 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:32,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:32,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 459 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397732977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:32,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397732977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:32,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:32,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:32,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:32,980 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:32,980 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:32,980 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:32,980 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:32,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:32,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397732981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,984 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,984 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:32,984 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,985 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:32,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:32,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b7b5f7f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:32,986 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:32,986 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,987 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,987 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:32,987 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:32,987 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:32,987 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,987 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:32,988 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,988 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:32,988 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:32,988 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:32,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:32,991 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:32,991 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:32,991 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:32,991 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:32,991 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:32,994 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,994 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:32,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,994 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:32,994 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,995 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:32,998 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,998 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:32,998 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:32,998 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:32,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:32,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:32,998 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:32,998 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,998 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:32,998 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:32,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:32,998 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:32,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:32,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:32,998 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:32,998 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:32,998 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:32,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:32,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,098 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:33,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:33,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:33,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:33,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:33,102 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:33,102 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,102 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:33,102 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:33,102 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,102 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,103 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,103 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,104 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,104 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:33,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:33,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:33,105 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:33,105 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,105 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:33,105 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:33,105 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:33,105 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,105 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:33,105 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:33,105 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,105 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:33,105 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:33,105 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,105 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:33,106 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:33,107 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:33,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6f94cb[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:33,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:33,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,111 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:33,111 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:33,111 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,111 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:33,111 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:33,112 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,112 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:33,112 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,112 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:33,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:33,115 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:33,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:33,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:33,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:33,115 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:33,116 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,116 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,116 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,117 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,118 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,120 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,120 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:33,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:33,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:33,120 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,120 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:33,120 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:33,121 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,121 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,121 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,121 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,121 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,121 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,121 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:33,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397733180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:33,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 461 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397733180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:33,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397733183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,221 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:33,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:33,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:33,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:33,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:33,224 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:33,224 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,224 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:33,224 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:33,225 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,225 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,227 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,228 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,228 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:33,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:33,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:33,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:33,229 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,229 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:33,229 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:33,229 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:33,230 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,231 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,231 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:33,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1112273d[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:33,232 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:33,232 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,233 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,233 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:33,234 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:33,233 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,234 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:33,234 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:33,235 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,235 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,235 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,235 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:33,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:33,238 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:33,238 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:33,238 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:33,238 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:33,238 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:33,241 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413ed4a77ae3bd3471a89d88c4cee47029b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ed4a77ae3bd3471a89d88c4cee47029b 2023-04-13 14:54:33,242 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,242 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,243 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,243 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,245 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,247 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:33,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:33,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,248 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:33,248 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:33,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:33,348 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:33,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:33,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:33,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:33,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:33,351 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:33,351 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,351 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:33,351 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:33,351 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,351 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,352 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,352 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,353 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,353 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:33,354 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:33,354 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,354 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:33,355 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,355 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:33,355 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:33,355 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,356 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,357 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,357 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:33,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ca1a017[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:33,358 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:33,358 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,363 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,363 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:33,363 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:33,363 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,363 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:33,363 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:33,365 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:33,365 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,365 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,365 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:33,367 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:33,367 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:33,367 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:33,367 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:33,367 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:33,368 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,368 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,369 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,369 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,373 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,373 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,373 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,373 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,373 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:33,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:33,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,373 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,373 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,373 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,373 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:33,374 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:33,373 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,374 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:33,474 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:33,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:33,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:33,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:33,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:33,477 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:33,477 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,477 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:33,477 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:33,477 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,478 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,479 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,479 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,479 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,479 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:33,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:33,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:33,480 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:33,480 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,480 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:33,480 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:33,480 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:33,480 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,480 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:33,481 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:33,481 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,481 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:33,481 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:33,481 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,481 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:33,481 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:33,483 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,483 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,483 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,483 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:33,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d208a5b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:33,484 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:33,485 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,486 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,486 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:33,486 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:33,486 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:33,486 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,486 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,488 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:33,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397733485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:33,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397733486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:33,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 463 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397733486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:33,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:33,491 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:33,491 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:33,491 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:33,491 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:33,491 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:33,492 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,493 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,493 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,493 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,494 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,497 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:33,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,498 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,498 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,498 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,498 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:33,498 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,498 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:33,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,598 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:33,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:33,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:33,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:33,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:33,601 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:33,601 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,601 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:33,601 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:33,601 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,601 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,603 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,603 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,603 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,603 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:33,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:33,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:33,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:33,604 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:33,604 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,604 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:33,604 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:33,604 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:33,604 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,604 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:33,604 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:33,604 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,604 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:33,605 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:33,605 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,605 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:33,605 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:33,607 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:33,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7efa82f8[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:33,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:33,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,610 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:33,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:33,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:33,610 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,610 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:33,611 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,611 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:33,611 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,611 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:33,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:33,614 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:33,614 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:33,614 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:33,614 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:33,614 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:33,615 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,616 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,616 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,616 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,617 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,620 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,620 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,620 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,620 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:33,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:33,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:33,620 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,620 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,620 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,620 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,620 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,621 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:33,620 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,621 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,651 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1272, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/10bd4c3b52de40df8528117325800908 2023-04-13 14:54:33,721 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:33,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:33,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:33,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:33,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:33,724 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:33,724 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,724 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:33,724 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:33,724 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,724 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,725 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,725 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,726 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,726 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:33,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:33,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:33,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:33,727 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,727 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:33,727 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:33,727 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:33,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:33,729 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,729 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,729 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,729 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:33,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ed169d5[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:33,731 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:33,731 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,732 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,732 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:33,732 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:33,732 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:33,732 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,732 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:33,737 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,737 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:33,737 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,737 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:33,739 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:33,739 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:33,739 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:33,739 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:33,739 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:33,743 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,743 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,743 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,743 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,744 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,747 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,747 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,747 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,747 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,747 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,747 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:33,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:33,747 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,747 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,747 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,747 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:33,747 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:33,747 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:33,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,847 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:33,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:33,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:33,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:33,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:33,851 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:33,851 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,851 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:33,851 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:33,851 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,851 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,852 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,853 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,853 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:33,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:33,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:33,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:33,854 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:33,854 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,854 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:33,854 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:33,854 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:33,854 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,854 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:33,854 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:33,854 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,854 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:33,854 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:33,854 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,855 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:33,855 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:33,858 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,858 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,859 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,859 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:33,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f11df00[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:33,860 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:33,860 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,861 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,861 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:33,861 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:33,861 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,862 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:33,862 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:33,863 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:33,863 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,863 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,863 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:33,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:33,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:33,865 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:33,865 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:33,865 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:33,865 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:33,865 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:33,867 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,867 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,867 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,868 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,869 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:33,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:33,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:33,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,872 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:33,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,872 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,872 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:33,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,873 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,973 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:33,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:33,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:33,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:33,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:33,976 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:33,976 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,976 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:33,976 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:33,976 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,976 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,978 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,978 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,978 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:33,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:33,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:33,979 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:33,979 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:33,979 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:33,979 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:33,979 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:33,979 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,980 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:33,980 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:33,980 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:33,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:33,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:33,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:33,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:33,981 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,981 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,981 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,982 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:33,983 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:33,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@760afceb[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:33,983 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,984 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,984 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:33,984 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:33,984 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,984 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:33,984 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:33,987 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:33,987 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,987 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,987 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:33,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:33,989 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:33,989 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:33,989 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:33,990 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:33,990 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:33,991 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,991 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,991 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:33,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:33,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:33,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397733990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:33,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:33,992 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:33,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,993 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:33,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:33,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397733992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:33,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 465 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397733992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:33,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,996 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,996 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,996 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:33,996 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:33,996 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:33,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:33,996 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:33,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:33,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:33,996 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:33,996 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:33,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:33,996 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:33,996 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:33,996 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:33,996 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:33,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:33,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:34,063 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1272 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/cdb5db12811a40c3a48224621b8a9d83 2023-04-13 14:54:34,096 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:34,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:34,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:34,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:34,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:34,099 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:34,100 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:34,100 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:34,100 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:34,100 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,100 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,101 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,101 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:34,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:34,102 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,102 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:34,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:34,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:34,103 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:34,103 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:34,103 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:34,103 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:34,103 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:34,103 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:34,103 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:34,103 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:34,103 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:34,103 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:34,103 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:34,104 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,104 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:34,104 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:34,106 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,106 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:34,107 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,107 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:34,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:34,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4d5d90[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:34,108 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:34,108 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:34,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:34,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:34,110 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,110 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:34,111 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,111 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:34,111 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:34,111 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:34,113 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:34,113 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:34,113 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:34,113 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:34,113 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:34,115 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,115 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:34,115 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:34,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:34,116 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,122 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,122 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:34,123 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:34,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:34,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:34,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:34,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:34,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:34,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:34,123 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:34,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:34,123 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:34,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:34,223 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:34,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:34,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:34,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:34,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:34,226 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:34,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:34,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:34,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:34,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,229 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,229 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:34,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:34,229 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,229 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:34,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:34,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:34,230 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:34,230 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:34,230 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:34,230 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:34,230 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:34,230 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:34,230 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:34,231 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:34,231 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:34,231 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:34,231 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:34,231 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,231 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:34,231 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:34,232 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,232 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:34,232 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,232 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:34,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:34,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2927180d[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:34,234 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:34,234 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,237 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,237 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:34,237 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:34,237 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:34,237 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,237 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:34,239 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:34,239 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:34,239 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,239 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:34,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:34,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:34,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:34,241 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:34,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:34,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:34,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:34,241 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:34,242 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,242 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:34,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,243 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,243 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:34,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:34,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,246 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:34,246 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:34,246 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:34,246 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:34,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:34,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:34,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:34,246 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,247 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:34,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:34,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:34,247 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:34,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:34,347 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:34,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:34,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:34,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:34,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:34,350 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:34,350 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:34,350 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:34,350 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:34,351 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,351 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,354 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:34,354 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:34,354 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,354 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:34,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:34,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:34,355 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:34,355 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:34,355 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:34,356 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:34,356 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:34,356 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:34,356 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,356 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:34,356 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:34,358 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,358 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:34,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:34,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:34,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@385f2bf6[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:34,360 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:34,360 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,363 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,363 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,363 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:34,363 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:34,363 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:34,363 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:34,366 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:34,366 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:34,366 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,366 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:34,369 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:34,369 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:34,369 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:34,369 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:34,369 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:34,371 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,371 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:34,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,371 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:34,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:34,372 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,373 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:34,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,377 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,377 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,377 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,377 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:34,377 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:34,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:34,377 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:34,377 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:34,377 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:34,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:34,377 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:34,377 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:34,377 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:34,377 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:34,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1272 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/a9432c6393b64b14b508f1870885f409 2023-04-13 14:54:34,477 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:34,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:34,480 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/10bd4c3b52de40df8528117325800908 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/10bd4c3b52de40df8528117325800908 2023-04-13 14:54:34,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:34,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:34,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:34,481 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:34,481 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:34,481 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:34,481 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:34,482 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,482 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,483 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:34,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:34,483 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,484 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,484 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:34,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:34,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:34,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:34,485 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:34,485 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:34,485 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:34,485 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:34,486 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:34,486 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:34,486 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:34,486 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:34,488 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,488 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:34,488 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/10bd4c3b52de40df8528117325800908, entries=400, sequenceid=1272, filesize=59.8 K 2023-04-13 14:54:34,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/cdb5db12811a40c3a48224621b8a9d83 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/cdb5db12811a40c3a48224621b8a9d83 2023-04-13 14:54:34,491 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,491 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@199375b2[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:34,493 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:34,493 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,494 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/cdb5db12811a40c3a48224621b8a9d83, entries=150, sequenceid=1272, filesize=11.9 K 2023-04-13 14:54:34,495 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,495 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:34,495 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:34,495 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:34,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/a9432c6393b64b14b508f1870885f409 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/a9432c6393b64b14b508f1870885f409 2023-04-13 14:54:34,496 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,496 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:34,498 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:34,498 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:34,498 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:34,498 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:34,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:34,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:34,503 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:34,503 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:34,503 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:34,503 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:34,503 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:34,504 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/a9432c6393b64b14b508f1870885f409, entries=150, sequenceid=1272, filesize=11.9 K 2023-04-13 14:54:34,505 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,505 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:34,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,505 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:34,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:34,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:34,506 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 0b6ed6780bbc11018cb686e788422bc1 in 1700ms, sequenceid=1272, compaction requested=true 2023-04-13 14:54:34,506 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:34,506 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,506 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 14:54:34,506 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:34,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,507 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 14:54:34,507 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:34,508 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:34,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:34,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,509 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 191894 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 14:54:34,509 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0b6ed6780bbc11018cb686e788422bc1/A is initiating minor compaction (all files) 2023-04-13 14:54:34,509 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b6ed6780bbc11018cb686e788422bc1/A in TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:34,509 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/54f8b8735dab426d8aa318291e03cba3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/c988bab369834edcae9483b7c44f8a00, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/d41d29abd2da4f6c9b50d5b316ba7f25, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/d4f07cf0e84f44058e996683914d3e7c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/3bdff373e70e4ab69df4458aef07bca3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/10bd4c3b52de40df8528117325800908] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp, totalSize=187.4 K 2023-04-13 14:54:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:34,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,511 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 54f8b8735dab426d8aa318291e03cba3, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=1188, earliestPutTs=1681397666662 2023-04-13 14:54:34,511 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73526 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 14:54:34,512 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0b6ed6780bbc11018cb686e788422bc1/B is initiating minor compaction (all files) 2023-04-13 14:54:34,512 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b6ed6780bbc11018cb686e788422bc1/B in TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:34,512 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/b619b7ed2ab94947842cf1cc8ecb58c8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/6f8c4cd1af054616a6c73b0914d57fea, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/931282876f404e948c7e000b1bb10d25, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/00866c4803604804be9fab74d4150167, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/44a18a1f9c774f5e99c67a63965e8e6c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/cdb5db12811a40c3a48224621b8a9d83] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp, totalSize=71.8 K 2023-04-13 14:54:34,512 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting c988bab369834edcae9483b7c44f8a00, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1213, earliestPutTs=1681397666721 2023-04-13 14:54:34,512 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting b619b7ed2ab94947842cf1cc8ecb58c8, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=1188, earliestPutTs=1681397666662 2023-04-13 14:54:34,513 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting d41d29abd2da4f6c9b50d5b316ba7f25, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1228, earliestPutTs=1681397667845 2023-04-13 14:54:34,513 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 6f8c4cd1af054616a6c73b0914d57fea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1213, earliestPutTs=1681397666721 2023-04-13 14:54:34,513 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting d4f07cf0e84f44058e996683914d3e7c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1248, earliestPutTs=1681397669989 2023-04-13 14:54:34,513 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 931282876f404e948c7e000b1bb10d25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1228, earliestPutTs=1681397667845 2023-04-13 14:54:34,514 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 3bdff373e70e4ab69df4458aef07bca3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1260, earliestPutTs=1681397670605 2023-04-13 14:54:34,514 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 00866c4803604804be9fab74d4150167, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1248, earliestPutTs=1681397669989 2023-04-13 14:54:34,514 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 10bd4c3b52de40df8528117325800908, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=1272, earliestPutTs=1681397672766 2023-04-13 14:54:34,514 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 44a18a1f9c774f5e99c67a63965e8e6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1260, earliestPutTs=1681397670605 2023-04-13 14:54:34,515 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting cdb5db12811a40c3a48224621b8a9d83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1272, earliestPutTs=1681397672802 2023-04-13 14:54:34,516 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:34,516 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:34,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:34,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:34,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:34,517 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:34,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:34,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:34,518 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:34,518 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:34,518 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,518 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,518 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,518 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,518 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,518 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:34,518 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,535 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b6ed6780bbc11018cb686e788422bc1#B#compaction#164 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:34,541 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b6ed6780bbc11018cb686e788422bc1#A#compaction#165 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:34,553 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/5e5b569fe9224e759de9c1e8848e8464 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/5e5b569fe9224e759de9c1e8848e8464 2023-04-13 14:54:34,558 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 0b6ed6780bbc11018cb686e788422bc1/B of 0b6ed6780bbc11018cb686e788422bc1 into 5e5b569fe9224e759de9c1e8848e8464(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:34,558 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:34,558 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., storeName=0b6ed6780bbc11018cb686e788422bc1/B, priority=10, startTime=1681397674507; duration=0sec 2023-04-13 14:54:34,558 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:34,559 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 14:54:34,561 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73526 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 14:54:34,561 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0b6ed6780bbc11018cb686e788422bc1/C is initiating minor compaction (all files) 2023-04-13 14:54:34,561 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b6ed6780bbc11018cb686e788422bc1/C in TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:34,561 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/1c0f8311662044acaf07b63554399451, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/26f935bf27644a9c9271d00399326d79, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/688570deda46432c9f1be821f881c033, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/7a68818da0ce4e9bbcc2cfd92b15e98e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/b446bdcabb244adc81b83b8788a99b3b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/a9432c6393b64b14b508f1870885f409] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp, totalSize=71.8 K 2023-04-13 14:54:34,563 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 1c0f8311662044acaf07b63554399451, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=1188, earliestPutTs=1681397666662 2023-04-13 14:54:34,563 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 26f935bf27644a9c9271d00399326d79, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1213, earliestPutTs=1681397666721 2023-04-13 14:54:34,564 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 688570deda46432c9f1be821f881c033, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1228, earliestPutTs=1681397667845 2023-04-13 14:54:34,564 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 7a68818da0ce4e9bbcc2cfd92b15e98e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1248, earliestPutTs=1681397669989 2023-04-13 14:54:34,564 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting b446bdcabb244adc81b83b8788a99b3b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1260, earliestPutTs=1681397670605 2023-04-13 14:54:34,565 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting a9432c6393b64b14b508f1870885f409, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1272, earliestPutTs=1681397672802 2023-04-13 14:54:34,577 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b6ed6780bbc11018cb686e788422bc1#C#compaction#166 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:34,618 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:34,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:34,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:34,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:34,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:34,621 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:34,621 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:34,621 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:34,621 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:34,622 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,622 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,623 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:34,623 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:34,623 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:34,623 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:34,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:34,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:34,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:34,625 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:34,625 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:34,625 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:34,629 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:34,630 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:34,630 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:34,630 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:34,630 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=120.76 KB heapSize=335.77 KB 2023-04-13 14:54:34,630 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:34,630 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:34,630 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:34,630 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:34,630 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:34,630 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:34,975 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/155d7e0d801d402e99864309b633fb99 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/155d7e0d801d402e99864309b633fb99 2023-04-13 14:54:34,986 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 0b6ed6780bbc11018cb686e788422bc1/A of 0b6ed6780bbc11018cb686e788422bc1 into 155d7e0d801d402e99864309b633fb99(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:34,986 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:34,986 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., storeName=0b6ed6780bbc11018cb686e788422bc1/A, priority=10, startTime=1681397674506; duration=0sec 2023-04-13 14:54:34,986 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:34,992 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/f0dbee0076d9499b9bef6a6e3c05605e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/f0dbee0076d9499b9bef6a6e3c05605e 2023-04-13 14:54:34,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:34,997 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:34,997 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 0b6ed6780bbc11018cb686e788422bc1/C of 0b6ed6780bbc11018cb686e788422bc1 into f0dbee0076d9499b9bef6a6e3c05605e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:34,997 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:34,997 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., storeName=0b6ed6780bbc11018cb686e788422bc1/C, priority=10, startTime=1681397674508; duration=0sec 2023-04-13 14:54:34,997 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:35,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 469 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397735005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397735007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397735008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,051 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304137b966f9f461c40f2b89bba86839f8812 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137b966f9f461c40f2b89bba86839f8812 2023-04-13 14:54:35,057 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1306, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/f823136c2f4a44aea978f0a09effc1b3 2023-04-13 14:54:35,079 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1306 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/4ad0e1d6224c43c881863aad85e6567b 2023-04-13 14:54:35,090 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1306 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/a3ca4993b84544e4a2e0463944a26ab9 2023-04-13 14:54:35,094 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/f823136c2f4a44aea978f0a09effc1b3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/f823136c2f4a44aea978f0a09effc1b3 2023-04-13 14:54:35,098 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/f823136c2f4a44aea978f0a09effc1b3, entries=150, sequenceid=1306, filesize=25.4 K 2023-04-13 14:54:35,100 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/4ad0e1d6224c43c881863aad85e6567b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/4ad0e1d6224c43c881863aad85e6567b 2023-04-13 14:54:35,106 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/4ad0e1d6224c43c881863aad85e6567b, entries=150, sequenceid=1306, filesize=11.9 K 2023-04-13 14:54:35,108 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/a3ca4993b84544e4a2e0463944a26ab9 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/a3ca4993b84544e4a2e0463944a26ab9 2023-04-13 14:54:35,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 471 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397735109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397735111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397735111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,119 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/a3ca4993b84544e4a2e0463944a26ab9, entries=150, sequenceid=1306, filesize=11.9 K 2023-04-13 14:54:35,120 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~335.72 KB/343776, currentSize=60.38 KB/61830 for 0b6ed6780bbc11018cb686e788422bc1 in 490ms, sequenceid=1306, compaction requested=false 2023-04-13 14:54:35,120 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:35,120 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:35,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:35,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:35,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:35,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:35,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,122 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:35,123 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,123 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:35,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:35,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f841acd[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:35,126 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:35,126 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,129 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:35,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:35,129 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,129 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:35,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:35,130 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,130 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:35,130 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:35,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:35,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:35,130 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:35,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:35,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:35,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:35,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:35,133 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:35,133 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:35,133 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:35,133 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:35,133 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:35,135 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,135 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:35,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,135 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:35,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:35,135 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:35,136 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:35,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:35,139 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:35,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:35,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:35,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:35,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:35,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:35,139 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:35,139 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:35,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,140 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:35,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:35,239 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:35,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:35,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:35,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:35,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:35,243 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:35,243 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:35,243 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:35,243 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:35,243 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,243 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:35,245 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:35,245 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:35,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:35,246 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,246 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:35,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:35,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:35,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:35,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:35,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:35,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:35,247 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 14:54:35,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:35,247 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:35,247 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:35,247 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:35,247 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:35,247 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:35,248 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:35,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:35,248 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:35,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:35,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:35,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:35,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:35,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:35,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:35,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:35,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:35,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:35,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:35,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:35,250 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,250 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:35,250 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,250 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:35,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:35,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:35,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:35,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:35,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e728840[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:35,252 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:35,252 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,254 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,254 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:35,254 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:35,254 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:35,254 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,254 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:35,255 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:35,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:35,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:35,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:35,259 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:35,259 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:35,259 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:35,259 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:35,259 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:35,262 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:35,262 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,262 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:35,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:35,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:35,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,264 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:35,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,266 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,267 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413f7f43afc94c4409e965ecabbefbeef99 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f7f43afc94c4409e965ecabbefbeef99 2023-04-13 14:54:35,268 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,268 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:35,268 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:35,268 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:35,268 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:35,268 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:35,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:35,268 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,269 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,269 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,268 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:35,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:35,269 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:35,269 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:35,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:35,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41140 deadline: 1681397735283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41122 deadline: 1681397735284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 473 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397735313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397735315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 421 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397735316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,369 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:35,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:35,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:35,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:35,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:35,372 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:35,372 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:35,372 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:35,372 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:35,373 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,373 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:35,374 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:35,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:35,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:35,374 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:35,375 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,375 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:35,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:35,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:35,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:35,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:35,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:35,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:35,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:35,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:35,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:35,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:35,376 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:35,376 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. started... 2023-04-13 14:54:35,376 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. as already flushing 2023-04-13 14:54:35,376 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:35,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:35,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:35,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:35,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:35,378 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,378 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:35,378 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,379 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:35,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:35,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:35,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:35,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:35,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f3314e3[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:35,380 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:35,380 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,383 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:35,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:35,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:35,383 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,383 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:35,384 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:35,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:35,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:35,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:35,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:35,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41140 deadline: 1681397735385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,387 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:35,387 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:35,387 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:35,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:35,387 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:35,387 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:35,389 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,389 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:35,389 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:35,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41122 deadline: 1681397735388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:35,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:35,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:35,392 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,397 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,397 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:35,397 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:35,397 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:35,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:35,397 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:35,397 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:35,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:35,398 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:35,398 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:35,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:35,398 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:35,398 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:35,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:35,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:35,398 DEBUG [Thread-853] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:35,398 DEBUG [Thread-853] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:35,449 DEBUG [Thread-854] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x210d1008 to 127.0.0.1:62626 2023-04-13 14:54:35,449 DEBUG [Thread-854] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:35,451 DEBUG [Thread-856] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00c2598b to 127.0.0.1:62626 2023-04-13 14:54:35,451 DEBUG [Thread-856] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:35,453 DEBUG [Thread-860] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08795451 to 127.0.0.1:62626 2023-04-13 14:54:35,454 DEBUG [Thread-860] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:35,454 DEBUG [Thread-858] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x508246cc to 127.0.0.1:62626 2023-04-13 14:54:35,454 DEBUG [Thread-858] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:35,498 DEBUG [Thread-853] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:35,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:35,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41140 deadline: 1681397735588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41122 deadline: 1681397735591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 475 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397735615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 423 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397735617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397735618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,675 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1321, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/002e82cfcf2e409eaf7d339617d35fb4 2023-04-13 14:54:35,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41140 deadline: 1681397735891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:35,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:35,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41122 deadline: 1681397735892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:36,085 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1321 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/e46ae1a454754bbb878236d7f1191f98 2023-04-13 14:54:36,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:36,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 477 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1681397736116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:36,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:36,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 425 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41104 deadline: 1681397736119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:36,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:36,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41142 deadline: 1681397736122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:36,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:36,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41140 deadline: 1681397736394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:36,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:36,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41122 deadline: 1681397736395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:36,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1321 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/0a4bd9a057a24c7b8517f11ed67dcae7 2023-04-13 14:54:36,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/002e82cfcf2e409eaf7d339617d35fb4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/002e82cfcf2e409eaf7d339617d35fb4 2023-04-13 14:54:36,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/002e82cfcf2e409eaf7d339617d35fb4, entries=200, sequenceid=1321, filesize=32.3 K 2023-04-13 14:54:36,506 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/e46ae1a454754bbb878236d7f1191f98 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/e46ae1a454754bbb878236d7f1191f98 2023-04-13 14:54:36,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/e46ae1a454754bbb878236d7f1191f98, entries=150, sequenceid=1321, filesize=11.9 K 2023-04-13 14:54:36,511 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/0a4bd9a057a24c7b8517f11ed67dcae7 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0a4bd9a057a24c7b8517f11ed67dcae7 2023-04-13 14:54:36,516 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0a4bd9a057a24c7b8517f11ed67dcae7, entries=150, sequenceid=1321, filesize=11.9 K 2023-04-13 14:54:36,516 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 0b6ed6780bbc11018cb686e788422bc1 in 1269ms, sequenceid=1321, compaction requested=true 2023-04-13 14:54:36,517 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:36,517 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:36,517 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:36,517 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:36,517 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:36,517 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:36,518 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85792 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:36,518 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 0b6ed6780bbc11018cb686e788422bc1/A is initiating minor compaction (all files) 2023-04-13 14:54:36,518 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b6ed6780bbc11018cb686e788422bc1/A in TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:36,518 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:36,518 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/155d7e0d801d402e99864309b633fb99, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/f823136c2f4a44aea978f0a09effc1b3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/002e82cfcf2e409eaf7d339617d35fb4] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp, totalSize=83.8 K 2023-04-13 14:54:36,518 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0b6ed6780bbc11018cb686e788422bc1/B is initiating minor compaction (all files) 2023-04-13 14:54:36,518 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b6ed6780bbc11018cb686e788422bc1/B in TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:36,518 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/5e5b569fe9224e759de9c1e8848e8464, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/4ad0e1d6224c43c881863aad85e6567b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/e46ae1a454754bbb878236d7f1191f98] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp, totalSize=36.3 K 2023-04-13 14:54:36,518 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 155d7e0d801d402e99864309b633fb99, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=1272, earliestPutTs=1681397672802 2023-04-13 14:54:36,519 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 5e5b569fe9224e759de9c1e8848e8464, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1272, earliestPutTs=1681397672802 2023-04-13 14:54:36,519 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting f823136c2f4a44aea978f0a09effc1b3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1306, earliestPutTs=1681397672862 2023-04-13 14:54:36,519 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 4ad0e1d6224c43c881863aad85e6567b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1306, earliestPutTs=1681397672862 2023-04-13 14:54:36,519 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 002e82cfcf2e409eaf7d339617d35fb4, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1321, earliestPutTs=1681397675004 2023-04-13 14:54:36,520 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting e46ae1a454754bbb878236d7f1191f98, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1321, earliestPutTs=1681397675004 2023-04-13 14:54:36,527 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b6ed6780bbc11018cb686e788422bc1#B#compaction#173 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:36,533 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b6ed6780bbc11018cb686e788422bc1#A#compaction#174 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:36,940 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/cf57594b0eab429d943b0c518c940da2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/cf57594b0eab429d943b0c518c940da2 2023-04-13 14:54:36,946 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b6ed6780bbc11018cb686e788422bc1/B of 0b6ed6780bbc11018cb686e788422bc1 into cf57594b0eab429d943b0c518c940da2(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:36,946 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:36,946 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., storeName=0b6ed6780bbc11018cb686e788422bc1/B, priority=13, startTime=1681397676517; duration=0sec 2023-04-13 14:54:36,946 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:36,946 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:36,947 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:36,947 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 0b6ed6780bbc11018cb686e788422bc1/C is initiating minor compaction (all files) 2023-04-13 14:54:36,947 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b6ed6780bbc11018cb686e788422bc1/C in TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:36,947 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/f0dbee0076d9499b9bef6a6e3c05605e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/a3ca4993b84544e4a2e0463944a26ab9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0a4bd9a057a24c7b8517f11ed67dcae7] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp, totalSize=36.3 K 2023-04-13 14:54:36,948 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting f0dbee0076d9499b9bef6a6e3c05605e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1272, earliestPutTs=1681397672802 2023-04-13 14:54:36,948 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting a3ca4993b84544e4a2e0463944a26ab9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1306, earliestPutTs=1681397672862 2023-04-13 14:54:36,948 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 0a4bd9a057a24c7b8517f11ed67dcae7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1321, earliestPutTs=1681397675004 2023-04-13 14:54:36,953 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b6ed6780bbc11018cb686e788422bc1#C#compaction#175 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:37,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:37,126 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 14:54:37,126 DEBUG [Thread-845] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x55cbb115 to 127.0.0.1:62626 2023-04-13 14:54:37,126 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:37,126 DEBUG [Thread-845] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:37,126 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:37,126 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:37,126 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:37,127 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:37,127 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:37,128 DEBUG [Thread-849] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24bd3872 to 127.0.0.1:62626 2023-04-13 14:54:37,128 DEBUG [Thread-849] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:37,131 DEBUG [Thread-851] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7aba8e46 to 127.0.0.1:62626 2023-04-13 14:54:37,131 DEBUG [Thread-851] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:37,353 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/b3129df0b2164dfeaa7715e2e1ad632f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/b3129df0b2164dfeaa7715e2e1ad632f 2023-04-13 14:54:37,358 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b6ed6780bbc11018cb686e788422bc1/A of 0b6ed6780bbc11018cb686e788422bc1 into b3129df0b2164dfeaa7715e2e1ad632f(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:37,358 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:37,358 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., storeName=0b6ed6780bbc11018cb686e788422bc1/A, priority=13, startTime=1681397676517; duration=0sec 2023-04-13 14:54:37,358 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:37,361 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/ba07092b812242e0a2728e4587281300 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/ba07092b812242e0a2728e4587281300 2023-04-13 14:54:37,365 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b6ed6780bbc11018cb686e788422bc1/C of 0b6ed6780bbc11018cb686e788422bc1 into ba07092b812242e0a2728e4587281300(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:37,365 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:37,365 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1., storeName=0b6ed6780bbc11018cb686e788422bc1/C, priority=13, startTime=1681397676517; duration=0sec 2023-04-13 14:54:37,365 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:37,397 DEBUG [Thread-847] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a71edf0 to 127.0.0.1:62626 2023-04-13 14:54:37,397 DEBUG [Thread-847] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:37,400 DEBUG [Thread-843] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x399105f2 to 127.0.0.1:62626 2023-04-13 14:54:37,400 DEBUG [Thread-843] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:37,400 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 14:54:37,400 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 175 2023-04-13 14:54:37,400 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 360 2023-04-13 14:54:37,400 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 170 2023-04-13 14:54:37,400 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 302 2023-04-13 14:54:37,400 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 267 2023-04-13 14:54:37,400 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 14:54:37,401 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(398): read 4792 2023-04-13 14:54:37,401 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(398): read 4864 2023-04-13 14:54:37,401 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 14:54:37,401 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(402): scanned 2130 2023-04-13 14:54:37,401 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(403): verified 6390 rows 2023-04-13 14:54:37,401 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(402): scanned 2143 2023-04-13 14:54:37,401 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(403): verified 6429 rows 2023-04-13 14:54:37,401 INFO [Listener at localhost/37101] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 14:54:37,401 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a607873 to 127.0.0.1:62626 2023-04-13 14:54:37,401 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:54:37,403 INFO [Listener at localhost/37101] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 14:54:37,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 14:54:37,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:37,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 14:54:37,409 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397677409"}]},"ts":"1681397677409"} 2023-04-13 14:54:37,410 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 14:54:37,411 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 14:54:37,411 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b6ed6780bbc11018cb686e788422bc1, UNASSIGN}] 2023-04-13 14:54:37,413 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b6ed6780bbc11018cb686e788422bc1, UNASSIGN 2023-04-13 14:54:37,413 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=0b6ed6780bbc11018cb686e788422bc1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:37,413 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397677413"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397677413"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397677413"}]},"ts":"1681397677413"} 2023-04-13 14:54:37,414 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:54:37,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 14:54:37,542 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304132c6995121e314495902446c065a57d38 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304132c6995121e314495902446c065a57d38 2023-04-13 14:54:37,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:37,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0b6ed6780bbc11018cb686e788422bc1, disabling compactions & flushes 2023-04-13 14:54:37,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:37,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 14:54:37,949 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1342, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/acefec7ad5f94585b6cc01b5f7e33829 2023-04-13 14:54:38,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 14:54:38,360 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1342 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/7e8c8ff7df30464d9a3af6a11c6aba2d 2023-04-13 14:54:38,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 14:54:38,769 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1342 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/7cdb0c68cd1f4784b437047a42faf4d5 2023-04-13 14:54:38,773 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/acefec7ad5f94585b6cc01b5f7e33829 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/acefec7ad5f94585b6cc01b5f7e33829 2023-04-13 14:54:38,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/acefec7ad5f94585b6cc01b5f7e33829, entries=150, sequenceid=1342, filesize=25.4 K 2023-04-13 14:54:38,777 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/7e8c8ff7df30464d9a3af6a11c6aba2d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/7e8c8ff7df30464d9a3af6a11c6aba2d 2023-04-13 14:54:38,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/7e8c8ff7df30464d9a3af6a11c6aba2d, entries=150, sequenceid=1342, filesize=11.9 K 2023-04-13 14:54:38,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/7cdb0c68cd1f4784b437047a42faf4d5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/7cdb0c68cd1f4784b437047a42faf4d5 2023-04-13 14:54:38,785 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/7cdb0c68cd1f4784b437047a42faf4d5, entries=150, sequenceid=1342, filesize=11.9 K 2023-04-13 14:54:38,786 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=26.84 KB/27480 for 0b6ed6780bbc11018cb686e788422bc1 in 1660ms, sequenceid=1342, compaction requested=false 2023-04-13 14:54:38,786 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:38,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:38,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:38,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. after waiting 0 ms 2023-04-13 14:54:38,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:38,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 0b6ed6780bbc11018cb686e788422bc1 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 14:54:38,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=A 2023-04-13 14:54:38,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:38,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=B 2023-04-13 14:54:38,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:38,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b6ed6780bbc11018cb686e788422bc1, store=C 2023-04-13 14:54:38,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:38,940 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-13 14:54:39,199 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413883c1a0f0df348f385e19bd565fd54f2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413883c1a0f0df348f385e19bd565fd54f2 2023-04-13 14:54:39,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 14:54:39,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1351, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/cc82e42d45394186bc37ebae94dc34cb 2023-04-13 14:54:40,013 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1351 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/50881cb8cd1f44ef83581ef207aeaf02 2023-04-13 14:54:40,434 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1351 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/97d97e6621ba4245b16dc5f217ef9308 2023-04-13 14:54:40,438 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/A/cc82e42d45394186bc37ebae94dc34cb as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/cc82e42d45394186bc37ebae94dc34cb 2023-04-13 14:54:40,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/cc82e42d45394186bc37ebae94dc34cb, entries=100, sequenceid=1351, filesize=18.6 K 2023-04-13 14:54:40,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/B/50881cb8cd1f44ef83581ef207aeaf02 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/50881cb8cd1f44ef83581ef207aeaf02 2023-04-13 14:54:40,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/50881cb8cd1f44ef83581ef207aeaf02, entries=100, sequenceid=1351, filesize=9.5 K 2023-04-13 14:54:40,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/.tmp/C/97d97e6621ba4245b16dc5f217ef9308 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/97d97e6621ba4245b16dc5f217ef9308 2023-04-13 14:54:40,451 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/97d97e6621ba4245b16dc5f217ef9308, entries=100, sequenceid=1351, filesize=9.5 K 2023-04-13 14:54:40,451 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 0b6ed6780bbc11018cb686e788422bc1 in 1665ms, sequenceid=1351, compaction requested=true 2023-04-13 14:54:40,472 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/207c612f0cfd4417940d69c45e3291d5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/02c4316e314b4a9396f612105c6b83e2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/91bc96a269b54e1f9d0622b13e4fde8f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/55c72473efd949b6955de42567a861ce, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/88f993cdb8bf41e18685fb8d51663b27, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/b25a91f98dac4d9c8a0292b5f01d5360, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/455ff6ca5b414298966ff23a38f44375, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/a0ae1102d4f647269bef0ae7bb8e57d6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/687bc298888341ff8231cc0d6e3a41f4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/7e30f17143b348e98d3380d72824da1a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/449f73381b3449b496b6ec72a611289f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/0792d828a74244a39b18d6cad9a3536f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/54f8b8735dab426d8aa318291e03cba3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/c988bab369834edcae9483b7c44f8a00, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/d41d29abd2da4f6c9b50d5b316ba7f25, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/d4f07cf0e84f44058e996683914d3e7c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/3bdff373e70e4ab69df4458aef07bca3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/10bd4c3b52de40df8528117325800908, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/155d7e0d801d402e99864309b633fb99, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/f823136c2f4a44aea978f0a09effc1b3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/002e82cfcf2e409eaf7d339617d35fb4] to archive 2023-04-13 14:54:40,473 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:54:40,475 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/207c612f0cfd4417940d69c45e3291d5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/207c612f0cfd4417940d69c45e3291d5 2023-04-13 14:54:40,476 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/02c4316e314b4a9396f612105c6b83e2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/02c4316e314b4a9396f612105c6b83e2 2023-04-13 14:54:40,477 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/91bc96a269b54e1f9d0622b13e4fde8f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/91bc96a269b54e1f9d0622b13e4fde8f 2023-04-13 14:54:40,478 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/55c72473efd949b6955de42567a861ce to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/55c72473efd949b6955de42567a861ce 2023-04-13 14:54:40,479 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/88f993cdb8bf41e18685fb8d51663b27 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/88f993cdb8bf41e18685fb8d51663b27 2023-04-13 14:54:40,480 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/b25a91f98dac4d9c8a0292b5f01d5360 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/b25a91f98dac4d9c8a0292b5f01d5360 2023-04-13 14:54:40,481 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/455ff6ca5b414298966ff23a38f44375 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/455ff6ca5b414298966ff23a38f44375 2023-04-13 14:54:40,482 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/a0ae1102d4f647269bef0ae7bb8e57d6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/a0ae1102d4f647269bef0ae7bb8e57d6 2023-04-13 14:54:40,483 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/687bc298888341ff8231cc0d6e3a41f4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/687bc298888341ff8231cc0d6e3a41f4 2023-04-13 14:54:40,484 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/7e30f17143b348e98d3380d72824da1a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/7e30f17143b348e98d3380d72824da1a 2023-04-13 14:54:40,485 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/449f73381b3449b496b6ec72a611289f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/449f73381b3449b496b6ec72a611289f 2023-04-13 14:54:40,486 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/0792d828a74244a39b18d6cad9a3536f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/0792d828a74244a39b18d6cad9a3536f 2023-04-13 14:54:40,487 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/54f8b8735dab426d8aa318291e03cba3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/54f8b8735dab426d8aa318291e03cba3 2023-04-13 14:54:40,488 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/c988bab369834edcae9483b7c44f8a00 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/c988bab369834edcae9483b7c44f8a00 2023-04-13 14:54:40,489 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/d41d29abd2da4f6c9b50d5b316ba7f25 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/d41d29abd2da4f6c9b50d5b316ba7f25 2023-04-13 14:54:40,490 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/d4f07cf0e84f44058e996683914d3e7c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/d4f07cf0e84f44058e996683914d3e7c 2023-04-13 14:54:40,491 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/3bdff373e70e4ab69df4458aef07bca3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/3bdff373e70e4ab69df4458aef07bca3 2023-04-13 14:54:40,492 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/10bd4c3b52de40df8528117325800908 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/10bd4c3b52de40df8528117325800908 2023-04-13 14:54:40,493 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/155d7e0d801d402e99864309b633fb99 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/155d7e0d801d402e99864309b633fb99 2023-04-13 14:54:40,494 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/f823136c2f4a44aea978f0a09effc1b3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/f823136c2f4a44aea978f0a09effc1b3 2023-04-13 14:54:40,495 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/002e82cfcf2e409eaf7d339617d35fb4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/002e82cfcf2e409eaf7d339617d35fb4 2023-04-13 14:54:40,497 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/02a21700016c416c9b6c7a4879afb23f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/17e14893526744d29dcf5eb067e28254, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/7882b81f765642bfb6e2789220ad01da, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/a7b8c9d9041947b4ad4120b524bb7970, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/d070d8b6feaa4058a22841ad42a9b8bc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/2cc23bc72cfd4350b9267919d17b6733, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/a8c58d644e0e4b599f3b03f6abb1ac32, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/8d8707dafbbf48a4836cd937be434b49, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/e1ab5c327fdf45048f128ebe390772e7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/9c8a054c2acd464091af02d4e71201d5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/5d1c10d06d894e148d320ad736704bef, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/b619b7ed2ab94947842cf1cc8ecb58c8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/58e99f0c12c04c6ba437585f75c41bbb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/6f8c4cd1af054616a6c73b0914d57fea, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/931282876f404e948c7e000b1bb10d25, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/00866c4803604804be9fab74d4150167, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/44a18a1f9c774f5e99c67a63965e8e6c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/5e5b569fe9224e759de9c1e8848e8464, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/cdb5db12811a40c3a48224621b8a9d83, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/4ad0e1d6224c43c881863aad85e6567b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/e46ae1a454754bbb878236d7f1191f98] to archive 2023-04-13 14:54:40,498 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:54:40,499 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/02a21700016c416c9b6c7a4879afb23f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/02a21700016c416c9b6c7a4879afb23f 2023-04-13 14:54:40,500 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/17e14893526744d29dcf5eb067e28254 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/17e14893526744d29dcf5eb067e28254 2023-04-13 14:54:40,502 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/7882b81f765642bfb6e2789220ad01da to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/7882b81f765642bfb6e2789220ad01da 2023-04-13 14:54:40,503 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/a7b8c9d9041947b4ad4120b524bb7970 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/a7b8c9d9041947b4ad4120b524bb7970 2023-04-13 14:54:40,503 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/d070d8b6feaa4058a22841ad42a9b8bc to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/d070d8b6feaa4058a22841ad42a9b8bc 2023-04-13 14:54:40,504 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/2cc23bc72cfd4350b9267919d17b6733 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/2cc23bc72cfd4350b9267919d17b6733 2023-04-13 14:54:40,506 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/a8c58d644e0e4b599f3b03f6abb1ac32 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/a8c58d644e0e4b599f3b03f6abb1ac32 2023-04-13 14:54:40,507 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/8d8707dafbbf48a4836cd937be434b49 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/8d8707dafbbf48a4836cd937be434b49 2023-04-13 14:54:40,508 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/e1ab5c327fdf45048f128ebe390772e7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/e1ab5c327fdf45048f128ebe390772e7 2023-04-13 14:54:40,509 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/9c8a054c2acd464091af02d4e71201d5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/9c8a054c2acd464091af02d4e71201d5 2023-04-13 14:54:40,510 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/5d1c10d06d894e148d320ad736704bef to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/5d1c10d06d894e148d320ad736704bef 2023-04-13 14:54:40,511 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/b619b7ed2ab94947842cf1cc8ecb58c8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/b619b7ed2ab94947842cf1cc8ecb58c8 2023-04-13 14:54:40,511 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/58e99f0c12c04c6ba437585f75c41bbb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/58e99f0c12c04c6ba437585f75c41bbb 2023-04-13 14:54:40,512 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/6f8c4cd1af054616a6c73b0914d57fea to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/6f8c4cd1af054616a6c73b0914d57fea 2023-04-13 14:54:40,513 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/931282876f404e948c7e000b1bb10d25 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/931282876f404e948c7e000b1bb10d25 2023-04-13 14:54:40,514 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/00866c4803604804be9fab74d4150167 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/00866c4803604804be9fab74d4150167 2023-04-13 14:54:40,515 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/44a18a1f9c774f5e99c67a63965e8e6c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/44a18a1f9c774f5e99c67a63965e8e6c 2023-04-13 14:54:40,516 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/5e5b569fe9224e759de9c1e8848e8464 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/5e5b569fe9224e759de9c1e8848e8464 2023-04-13 14:54:40,517 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/cdb5db12811a40c3a48224621b8a9d83 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/cdb5db12811a40c3a48224621b8a9d83 2023-04-13 14:54:40,518 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/4ad0e1d6224c43c881863aad85e6567b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/4ad0e1d6224c43c881863aad85e6567b 2023-04-13 14:54:40,519 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/e46ae1a454754bbb878236d7f1191f98 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/e46ae1a454754bbb878236d7f1191f98 2023-04-13 14:54:40,521 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/19f01c9600ea49f8b800e4ae25590d0f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/73e0cde10bb748c5a49f99a365bb55f8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/32066e63404641b78e2341df8602a263, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0425b9852b24474ab5410a84a13f7d8c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/911cea0cca364293b05af43812b92ab2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/d4d2acd59cf24228925d7ba9a42d3ea1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/5cbef882dc52454ba07ab8c7e43466da, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/e91d9f575ac34da193f21aaec1f9e632, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/fa54aab2eea8475f848086c010e00807, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/607f9d6122da4459b8179645ba84bb68, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0bc6c1da9d2644cdbfc9424a03665591, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/1c0f8311662044acaf07b63554399451, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/8014d90b9a27451aa6aff695f2a0fab8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/26f935bf27644a9c9271d00399326d79, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/688570deda46432c9f1be821f881c033, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/7a68818da0ce4e9bbcc2cfd92b15e98e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/b446bdcabb244adc81b83b8788a99b3b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/f0dbee0076d9499b9bef6a6e3c05605e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/a9432c6393b64b14b508f1870885f409, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/a3ca4993b84544e4a2e0463944a26ab9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0a4bd9a057a24c7b8517f11ed67dcae7] to archive 2023-04-13 14:54:40,522 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:54:40,523 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/19f01c9600ea49f8b800e4ae25590d0f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/19f01c9600ea49f8b800e4ae25590d0f 2023-04-13 14:54:40,524 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/73e0cde10bb748c5a49f99a365bb55f8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/73e0cde10bb748c5a49f99a365bb55f8 2023-04-13 14:54:40,525 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/32066e63404641b78e2341df8602a263 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/32066e63404641b78e2341df8602a263 2023-04-13 14:54:40,526 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0425b9852b24474ab5410a84a13f7d8c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0425b9852b24474ab5410a84a13f7d8c 2023-04-13 14:54:40,527 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/911cea0cca364293b05af43812b92ab2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/911cea0cca364293b05af43812b92ab2 2023-04-13 14:54:40,528 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/d4d2acd59cf24228925d7ba9a42d3ea1 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/d4d2acd59cf24228925d7ba9a42d3ea1 2023-04-13 14:54:40,529 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/5cbef882dc52454ba07ab8c7e43466da to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/5cbef882dc52454ba07ab8c7e43466da 2023-04-13 14:54:40,530 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/e91d9f575ac34da193f21aaec1f9e632 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/e91d9f575ac34da193f21aaec1f9e632 2023-04-13 14:54:40,531 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/fa54aab2eea8475f848086c010e00807 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/fa54aab2eea8475f848086c010e00807 2023-04-13 14:54:40,532 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/607f9d6122da4459b8179645ba84bb68 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/607f9d6122da4459b8179645ba84bb68 2023-04-13 14:54:40,533 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0bc6c1da9d2644cdbfc9424a03665591 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0bc6c1da9d2644cdbfc9424a03665591 2023-04-13 14:54:40,534 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/1c0f8311662044acaf07b63554399451 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/1c0f8311662044acaf07b63554399451 2023-04-13 14:54:40,535 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/8014d90b9a27451aa6aff695f2a0fab8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/8014d90b9a27451aa6aff695f2a0fab8 2023-04-13 14:54:40,536 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/26f935bf27644a9c9271d00399326d79 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/26f935bf27644a9c9271d00399326d79 2023-04-13 14:54:40,537 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/688570deda46432c9f1be821f881c033 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/688570deda46432c9f1be821f881c033 2023-04-13 14:54:40,538 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/7a68818da0ce4e9bbcc2cfd92b15e98e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/7a68818da0ce4e9bbcc2cfd92b15e98e 2023-04-13 14:54:40,539 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/b446bdcabb244adc81b83b8788a99b3b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/b446bdcabb244adc81b83b8788a99b3b 2023-04-13 14:54:40,540 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/f0dbee0076d9499b9bef6a6e3c05605e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/f0dbee0076d9499b9bef6a6e3c05605e 2023-04-13 14:54:40,541 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/a9432c6393b64b14b508f1870885f409 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/a9432c6393b64b14b508f1870885f409 2023-04-13 14:54:40,542 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/a3ca4993b84544e4a2e0463944a26ab9 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/a3ca4993b84544e4a2e0463944a26ab9 2023-04-13 14:54:40,543 DEBUG [StoreCloser-TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0a4bd9a057a24c7b8517f11ed67dcae7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/0a4bd9a057a24c7b8517f11ed67dcae7 2023-04-13 14:54:40,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/recovered.edits/1354.seqid, newMaxSeqId=1354, maxSeqId=4 2023-04-13 14:54:40,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1. 2023-04-13 14:54:40,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0b6ed6780bbc11018cb686e788422bc1: 2023-04-13 14:54:40,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:40,550 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=0b6ed6780bbc11018cb686e788422bc1, regionState=CLOSED 2023-04-13 14:54:40,550 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397680550"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397680550"}]},"ts":"1681397680550"} 2023-04-13 14:54:40,552 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-13 14:54:40,552 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 0b6ed6780bbc11018cb686e788422bc1, server=jenkins-hbase4.apache.org,43875,1681397617638 in 3.1370 sec 2023-04-13 14:54:40,553 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-13 14:54:40,553 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b6ed6780bbc11018cb686e788422bc1, UNASSIGN in 3.1410 sec 2023-04-13 14:54:40,554 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397680554"}]},"ts":"1681397680554"} 2023-04-13 14:54:40,555 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 14:54:40,556 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 14:54:40,557 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.1540 sec 2023-04-13 14:54:41,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 14:54:41,514 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-13 14:54:41,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 14:54:41,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:41,517 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:41,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-13 14:54:41,518 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:41,522 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:41,525 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/recovered.edits] 2023-04-13 14:54:41,528 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/acefec7ad5f94585b6cc01b5f7e33829 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/acefec7ad5f94585b6cc01b5f7e33829 2023-04-13 14:54:41,529 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/b3129df0b2164dfeaa7715e2e1ad632f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/b3129df0b2164dfeaa7715e2e1ad632f 2023-04-13 14:54:41,530 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/cc82e42d45394186bc37ebae94dc34cb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/A/cc82e42d45394186bc37ebae94dc34cb 2023-04-13 14:54:41,533 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/50881cb8cd1f44ef83581ef207aeaf02 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/50881cb8cd1f44ef83581ef207aeaf02 2023-04-13 14:54:41,534 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/7e8c8ff7df30464d9a3af6a11c6aba2d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/7e8c8ff7df30464d9a3af6a11c6aba2d 2023-04-13 14:54:41,535 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/cf57594b0eab429d943b0c518c940da2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/B/cf57594b0eab429d943b0c518c940da2 2023-04-13 14:54:41,538 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/7cdb0c68cd1f4784b437047a42faf4d5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/7cdb0c68cd1f4784b437047a42faf4d5 2023-04-13 14:54:41,539 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/97d97e6621ba4245b16dc5f217ef9308 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/97d97e6621ba4245b16dc5f217ef9308 2023-04-13 14:54:41,541 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/ba07092b812242e0a2728e4587281300 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/C/ba07092b812242e0a2728e4587281300 2023-04-13 14:54:41,544 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/recovered.edits/1354.seqid to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1/recovered.edits/1354.seqid 2023-04-13 14:54:41,545 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/0b6ed6780bbc11018cb686e788422bc1 2023-04-13 14:54:41,545 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 14:54:41,546 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 14:54:41,546 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-13 14:54:41,550 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304132c6995121e314495902446c065a57d38 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304132c6995121e314495902446c065a57d38 2023-04-13 14:54:41,551 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041337a87df58ad94a0281839e62be9a8905 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041337a87df58ad94a0281839e62be9a8905 2023-04-13 14:54:41,552 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133927256efe8b4e17b388931d387e096b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133927256efe8b4e17b388931d387e096b 2023-04-13 14:54:41,553 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041350e12dee1708494abc562727ce5f845d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041350e12dee1708494abc562727ce5f845d 2023-04-13 14:54:41,554 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041356a604766fa2420d9784a436ca222151 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041356a604766fa2420d9784a436ca222151 2023-04-13 14:54:41,555 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136e6abd2cb6dd421aa27fcb6c36fc90b0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136e6abd2cb6dd421aa27fcb6c36fc90b0 2023-04-13 14:54:41,556 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041370390f46de1f407b8c1a010cc70c7ad0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041370390f46de1f407b8c1a010cc70c7ad0 2023-04-13 14:54:41,557 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137b966f9f461c40f2b89bba86839f8812 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137b966f9f461c40f2b89bba86839f8812 2023-04-13 14:54:41,558 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137bc06edc695b4fd29af4e1974f9e0efc to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137bc06edc695b4fd29af4e1974f9e0efc 2023-04-13 14:54:41,560 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041382b24851b39f4c4ea06c52fbfea5630b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041382b24851b39f4c4ea06c52fbfea5630b 2023-04-13 14:54:41,561 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413883c1a0f0df348f385e19bd565fd54f2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413883c1a0f0df348f385e19bd565fd54f2 2023-04-13 14:54:41,562 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304139ef3a22f53d04bc782999d9c11e5c027 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304139ef3a22f53d04bc782999d9c11e5c027 2023-04-13 14:54:41,563 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413a0536f5971b84ec4a34658f188698b10 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413a0536f5971b84ec4a34658f188698b10 2023-04-13 14:54:41,564 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b59cd31294664d85ba061c164125ca36 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b59cd31294664d85ba061c164125ca36 2023-04-13 14:54:41,565 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c6b25c7d165e4e07bcfa58af165a08d2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c6b25c7d165e4e07bcfa58af165a08d2 2023-04-13 14:54:41,566 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d3ada7ec4d5f46cd938cca2db8b8558a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d3ada7ec4d5f46cd938cca2db8b8558a 2023-04-13 14:54:41,567 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e2181ca0b4b64dbe874af3332e217ceb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e2181ca0b4b64dbe874af3332e217ceb 2023-04-13 14:54:41,568 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ed4a77ae3bd3471a89d88c4cee47029b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ed4a77ae3bd3471a89d88c4cee47029b 2023-04-13 14:54:41,569 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f7f43afc94c4409e965ecabbefbeef99 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f7f43afc94c4409e965ecabbefbeef99 2023-04-13 14:54:41,569 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 14:54:41,572 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:41,573 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 14:54:41,576 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 14:54:41,577 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:41,577 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 14:54:41,577 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681397681577"}]},"ts":"9223372036854775807"} 2023-04-13 14:54:41,578 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 14:54:41,578 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0b6ed6780bbc11018cb686e788422bc1, NAME => 'TestAcidGuarantees,,1681397653316.0b6ed6780bbc11018cb686e788422bc1.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 14:54:41,578 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 14:54:41,578 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681397681578"}]},"ts":"9223372036854775807"} 2023-04-13 14:54:41,580 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 14:54:41,582 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:41,583 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 67 msec 2023-04-13 14:54:41,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-13 14:54:41,619 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-13 14:54:41,633 INFO [Listener at localhost/37101] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=227 (was 231), OpenFileDescriptor=531 (was 536), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=356 (was 361), ProcessCount=169 (was 169), AvailableMemoryMB=814 (was 968) 2023-04-13 14:54:41,644 INFO [Listener at localhost/37101] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227, OpenFileDescriptor=531, MaxFileDescriptor=60000, SystemLoadAverage=356, ProcessCount=169, AvailableMemoryMB=814 2023-04-13 14:54:41,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 14:54:41,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 14:54:41,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 14:54:41,649 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 14:54:41,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-13 14:54:41,650 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 14:54:41,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 14:54:41,651 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:41,651 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e empty. 2023-04-13 14:54:41,652 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:41,652 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 14:54:41,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 14:54:41,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 14:54:42,060 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 14:54:42,062 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => ccd2cd4c855e3a8c5393dd57d5cc737e, NAME => 'TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp 2023-04-13 14:54:42,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 14:54:42,468 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:54:42,468 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing ccd2cd4c855e3a8c5393dd57d5cc737e, disabling compactions & flushes 2023-04-13 14:54:42,468 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:42,468 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:42,468 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. after waiting 0 ms 2023-04-13 14:54:42,468 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:42,468 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:42,468 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:42,470 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 14:54:42,471 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397682471"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397682471"}]},"ts":"1681397682471"} 2023-04-13 14:54:42,472 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 14:54:42,473 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 14:54:42,473 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397682473"}]},"ts":"1681397682473"} 2023-04-13 14:54:42,474 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 14:54:42,476 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ccd2cd4c855e3a8c5393dd57d5cc737e, ASSIGN}] 2023-04-13 14:54:42,478 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ccd2cd4c855e3a8c5393dd57d5cc737e, ASSIGN 2023-04-13 14:54:42,478 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ccd2cd4c855e3a8c5393dd57d5cc737e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43875,1681397617638; forceNewPlan=false, retain=false 2023-04-13 14:54:42,629 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=ccd2cd4c855e3a8c5393dd57d5cc737e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:42,630 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397682629"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397682629"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397682629"}]},"ts":"1681397682629"} 2023-04-13 14:54:42,631 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:54:42,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 14:54:42,787 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:42,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ccd2cd4c855e3a8c5393dd57d5cc737e, NAME => 'TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.', STARTKEY => '', ENDKEY => ''} 2023-04-13 14:54:42,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:42,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:54:42,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:42,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:42,789 INFO [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:42,791 DEBUG [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A 2023-04-13 14:54:42,791 DEBUG [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A 2023-04-13 14:54:42,791 INFO [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:54:42,791 INFO [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ccd2cd4c855e3a8c5393dd57d5cc737e columnFamilyName A 2023-04-13 14:54:42,792 INFO [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] regionserver.HStore(310): Store=ccd2cd4c855e3a8c5393dd57d5cc737e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:54:42,792 INFO [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:42,793 DEBUG [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B 2023-04-13 14:54:42,793 DEBUG [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B 2023-04-13 14:54:42,794 INFO [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:54:42,794 INFO [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ccd2cd4c855e3a8c5393dd57d5cc737e columnFamilyName B 2023-04-13 14:54:42,794 INFO [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] regionserver.HStore(310): Store=ccd2cd4c855e3a8c5393dd57d5cc737e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:54:42,794 INFO [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:42,795 DEBUG [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C 2023-04-13 14:54:42,795 DEBUG [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C 2023-04-13 14:54:42,796 INFO [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:54:42,796 INFO [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ccd2cd4c855e3a8c5393dd57d5cc737e columnFamilyName C 2023-04-13 14:54:42,796 INFO [StoreOpener-ccd2cd4c855e3a8c5393dd57d5cc737e-1] regionserver.HStore(310): Store=ccd2cd4c855e3a8c5393dd57d5cc737e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:54:42,796 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:42,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:42,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:42,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 14:54:42,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:42,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 14:54:42,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ccd2cd4c855e3a8c5393dd57d5cc737e; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69499911, jitterRate=0.03562937676906586}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 14:54:42,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:42,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., pid=30, masterSystemTime=1681397682782 2023-04-13 14:54:42,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:42,805 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:42,805 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=ccd2cd4c855e3a8c5393dd57d5cc737e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:42,806 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397682805"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681397682805"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681397682805"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681397682805"}]},"ts":"1681397682805"} 2023-04-13 14:54:42,808 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-13 14:54:42,808 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 in 175 msec 2023-04-13 14:54:42,809 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-13 14:54:42,809 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ccd2cd4c855e3a8c5393dd57d5cc737e, ASSIGN in 332 msec 2023-04-13 14:54:42,809 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 14:54:42,810 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397682809"}]},"ts":"1681397682809"} 2023-04-13 14:54:42,810 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 14:54:42,812 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 14:54:42,813 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1650 sec 2023-04-13 14:54:43,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 14:54:43,754 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-13 14:54:43,755 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x112921f2 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:43,761 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22b4eae5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:43,762 DEBUG [Listener at localhost/37101] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:43,764 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53204, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:43,765 DEBUG [Listener at localhost/37101] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 14:54:43,767 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53270, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 14:54:43,768 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x7ed1dde9 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:43,771 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@262f4570, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:43,772 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x70043cf0 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:43,775 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ca898c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:43,775 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x34b7fed0 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:43,779 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6585a70a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:43,779 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x7f4ad99d to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:43,785 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7480f9da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:43,785 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x330b77da to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:43,788 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a24d617, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:43,789 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x104dbf4e to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:43,792 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ce5573e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:43,792 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x753156b2 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:43,796 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@87b3095, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:43,796 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x6dea0f9b to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:43,801 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12115052, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:43,801 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x434e0eab to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:43,805 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e65578e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:43,805 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x5c43dddd to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:54:43,808 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbaba8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:54:43,813 DEBUG [hconnection-0x5d37ab42-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:43,813 DEBUG [hconnection-0xfe5b0ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:43,814 DEBUG [hconnection-0x627ced3e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:43,815 DEBUG [hconnection-0x46e3cbae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:43,815 DEBUG [hconnection-0x25e03b7a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:43,815 DEBUG [hconnection-0x751baaf6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:43,816 DEBUG [hconnection-0x6172ee28-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:43,816 DEBUG [hconnection-0x40e2596f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:43,817 DEBUG [hconnection-0x55e649dd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:43,817 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53232, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:43,817 DEBUG [hconnection-0x634f4ff4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:54:43,818 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53254, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:43,818 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53282, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:43,820 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53214, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:43,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:43,826 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53222, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:43,827 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53236, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:43,828 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53244, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:43,828 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53266, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:43,828 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53274, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:43,828 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53292, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:54:43,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:43,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:43,836 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:43,836 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:43,836 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:43,836 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:43,836 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:43,836 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:43,838 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:43,838 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:43,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:43,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:43,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:43,838 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:43,838 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:43,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:43,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:43,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:43,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:43,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:43,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:43,839 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:43,839 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:43,839 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:43,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:43,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:43,840 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:43,840 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:43,841 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 14:54:43,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:43,844 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:43,844 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:43,845 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:43,845 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:43,845 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:43,845 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:43,845 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:43,855 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/d1f0cc8e66954a20ab3a7584f370d796 2023-04-13 14:54:43,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397683858, max timestamp=1681397683862Num uniques 859; 2023-04-13 14:54:43,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397683858, max timestamp=1681397683862Num uniques 859; 2023-04-13 14:54:43,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:43,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:43,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397683859, max timestamp=1681397683862Num uniques 859; 2023-04-13 14:54:43,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:54:43,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:43,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397743875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:43,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:43,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397743876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:43,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:43,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397743877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:43,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:43,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397743878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:43,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:43,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397743878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:43,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:43,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397743980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:43,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:43,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:43,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397743981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:43,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397743981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:43,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:43,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397743983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:43,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:43,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397743983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397744184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397744186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397744186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397744186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397744186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,277 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/eff61f487c734a628c713b5f82b3ef4e 2023-04-13 14:54:44,292 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/86f19b1950b5419e90280b4a43013e62 2023-04-13 14:54:44,296 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/d1f0cc8e66954a20ab3a7584f370d796 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d1f0cc8e66954a20ab3a7584f370d796 2023-04-13 14:54:44,303 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d1f0cc8e66954a20ab3a7584f370d796, entries=150, sequenceid=11, filesize=11.6 K 2023-04-13 14:54:44,304 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/eff61f487c734a628c713b5f82b3ef4e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/eff61f487c734a628c713b5f82b3ef4e 2023-04-13 14:54:44,308 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/eff61f487c734a628c713b5f82b3ef4e, entries=150, sequenceid=11, filesize=11.6 K 2023-04-13 14:54:44,308 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/86f19b1950b5419e90280b4a43013e62 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/86f19b1950b5419e90280b4a43013e62 2023-04-13 14:54:44,317 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/86f19b1950b5419e90280b4a43013e62, entries=150, sequenceid=11, filesize=11.6 K 2023-04-13 14:54:44,318 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for ccd2cd4c855e3a8c5393dd57d5cc737e in 477ms, sequenceid=11, compaction requested=false 2023-04-13 14:54:44,318 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:44,318 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:44,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:44,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:44,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,320 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:44,320 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:44,321 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:44,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:44,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:44,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:44,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:44,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:44,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:44,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d0fc052[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:44,323 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:44,324 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:44,325 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,325 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:44,329 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:44,329 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:44,329 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:44,329 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:44,329 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:44,329 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:44,330 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:44,331 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:44,331 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,331 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,331 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:44,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:44,334 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:44,334 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:44,334 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:44,334 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:44,334 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:44,342 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:44,342 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:44,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:44,342 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:44,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:44,343 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:44,344 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:44,346 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,346 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:44,346 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:44,346 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,346 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:44,346 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:44,346 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:44,346 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:44,346 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:44,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:44,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:44,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:44,346 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:44,347 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:44,347 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:44,347 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:44,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:44,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:44,447 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:44,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:44,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:44,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:44,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:44,450 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:44,451 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:44,451 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:44,451 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:44,451 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:44,451 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:44,452 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:44,452 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:44,453 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,453 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:44,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:44,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:44,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:44,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:44,454 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:44,454 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:44,454 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:44,454 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:44,454 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:44,454 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:44,454 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:44,454 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=131.94 KB heapSize=375.22 KB 2023-04-13 14:54:44,455 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:44,455 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:44,455 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:44,455 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:44,455 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:44,455 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:44,464 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/bf1116504a6743d8b60fe9fc5a87bc31 2023-04-13 14:54:44,483 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/94d69f68cf7843948ea1cf87c671f744 2023-04-13 14:54:44,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:44,488 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:44,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397744492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397744492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397744514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397744514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397744514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397744615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397744615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397744618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397744618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397744618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397744819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397744819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397744825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:44,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397744825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397744825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,901 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/a39d90b3c7de4bc3ab9b878cb8b04520 2023-04-13 14:54:44,905 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/bf1116504a6743d8b60fe9fc5a87bc31 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/bf1116504a6743d8b60fe9fc5a87bc31 2023-04-13 14:54:44,909 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/bf1116504a6743d8b60fe9fc5a87bc31, entries=150, sequenceid=47, filesize=11.6 K 2023-04-13 14:54:44,910 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/94d69f68cf7843948ea1cf87c671f744 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/94d69f68cf7843948ea1cf87c671f744 2023-04-13 14:54:44,914 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/94d69f68cf7843948ea1cf87c671f744, entries=150, sequenceid=47, filesize=11.6 K 2023-04-13 14:54:44,915 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/a39d90b3c7de4bc3ab9b878cb8b04520 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/a39d90b3c7de4bc3ab9b878cb8b04520 2023-04-13 14:54:44,919 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/a39d90b3c7de4bc3ab9b878cb8b04520, entries=150, sequenceid=47, filesize=11.6 K 2023-04-13 14:54:44,920 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~375.17 KB/384176, currentSize=46.96 KB/48090 for ccd2cd4c855e3a8c5393dd57d5cc737e in 466ms, sequenceid=47, compaction requested=false 2023-04-13 14:54:44,920 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:44,920 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:44,920 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:44,920 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:44,920 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,920 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:44,920 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:44,921 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,921 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:44,922 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:44,922 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:44,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:44,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:44,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7252ef4c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:44,923 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:44,923 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:44,925 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:44,926 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,926 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:44,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:44,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:44,936 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:44,936 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:44,936 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:44,937 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:44,937 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,937 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:44,937 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:44,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:44,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:44,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:44,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:44,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:44,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:44,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:44,940 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:44,940 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:44,940 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:44,940 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:44,940 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:44,941 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:44,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:44,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:44,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:44,942 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:44,942 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:44,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:44,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:44,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:44,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:44,942 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:44,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,944 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:44,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:44,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,950 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,950 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:44,950 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:44,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:44,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:44,950 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:44,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:44,951 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:44,951 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:44,950 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:44,950 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:44,951 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:44,951 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:44,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:44,951 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:44,951 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,051 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:45,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:45,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:45,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:45,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:45,054 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:45,055 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:45,055 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:45,055 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:45,055 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,055 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,056 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:45,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:45,056 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,057 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,057 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:45,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:45,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:45,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:45,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:45,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:45,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:45,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:45,059 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:45,059 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:45,059 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:54:45,059 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:45,059 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:45,059 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:45,059 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:45,059 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:45,059 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:45,067 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/769c6646eb76421497b04bde9feaf211 2023-04-13 14:54:45,080 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/5538dbb055d544f39c7cbfa8212cb68b 2023-04-13 14:54:45,099 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/244f58b4b2b54dedab97704f2427d376 2023-04-13 14:54:45,103 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/769c6646eb76421497b04bde9feaf211 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/769c6646eb76421497b04bde9feaf211 2023-04-13 14:54:45,107 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/769c6646eb76421497b04bde9feaf211, entries=150, sequenceid=57, filesize=11.6 K 2023-04-13 14:54:45,108 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/5538dbb055d544f39c7cbfa8212cb68b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5538dbb055d544f39c7cbfa8212cb68b 2023-04-13 14:54:45,111 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5538dbb055d544f39c7cbfa8212cb68b, entries=150, sequenceid=57, filesize=11.6 K 2023-04-13 14:54:45,112 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/244f58b4b2b54dedab97704f2427d376 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/244f58b4b2b54dedab97704f2427d376 2023-04-13 14:54:45,118 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/244f58b4b2b54dedab97704f2427d376, entries=150, sequenceid=57, filesize=11.6 K 2023-04-13 14:54:45,119 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for ccd2cd4c855e3a8c5393dd57d5cc737e in 60ms, sequenceid=57, compaction requested=true 2023-04-13 14:54:45,119 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:45,119 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:45,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:45,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:45,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:45,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:45,121 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:45,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:45,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:45,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:45,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58c859ce[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:45,124 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:45,124 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,126 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:45,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:45,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:45,126 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,126 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:45,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:45,128 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:45,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:45,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:45,131 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:54:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:45,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:45,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:45,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:45,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:45,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:45,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:45,132 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:45,132 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:45,132 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:45,132 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:45,132 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:45,134 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,134 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:45,134 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:45,135 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,137 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:45,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:45,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:45,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:45,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:45,139 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:45,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,140 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:45,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:45,140 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:45,140 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:45,140 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,140 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,140 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,140 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,140 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397745149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397685146, max timestamp=1681397685151Num uniques 859; 2023-04-13 14:54:45,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:45,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397685146, max timestamp=1681397685151Num uniques 859; 2023-04-13 14:54:45,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:45,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681397685146, max timestamp=1681397685151Num uniques 859; 2023-04-13 14:54:45,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:45,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397745188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397745189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397745190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397745190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,240 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:45,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:45,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:45,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:45,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:45,244 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:45,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:45,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:45,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:45,245 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,245 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,246 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:45,246 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:45,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:45,247 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,247 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:45,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:45,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:45,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:45,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:45,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:45,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:45,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:45,249 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:45,249 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:45,249 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:45,249 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,249 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:45,249 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:45,251 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,251 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:45,251 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,251 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:45,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:45,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fbbda38[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:45,253 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:45,253 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,255 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:45,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:45,255 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,255 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:45,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:45,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397745254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,257 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:45,257 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:45,257 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:45,257 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:45,259 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:45,259 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:45,259 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:45,259 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:45,259 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:45,260 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,260 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:45,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,260 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:45,261 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:45,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,262 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:45,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:45,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:45,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:45,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:45,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:45,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,266 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:45,266 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:45,266 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:45,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:45,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:45,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:45,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397745293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397745293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397745294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397745294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,366 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:45,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:45,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:45,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:45,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:45,370 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:45,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:45,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:45,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:45,371 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,371 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,374 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:45,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:45,374 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:45,374 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,374 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:45,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:45,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:45,375 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:45,375 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:45,375 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:45,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:45,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:45,376 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:45,376 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:45,376 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:45,376 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:45,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:45,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:45,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:45,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:45,377 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:45,378 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,378 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:45,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:45,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cb580cd[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:45,379 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:45,379 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,380 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:45,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:45,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:45,380 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,380 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:45,382 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:45,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:45,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:45,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:45,385 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:45,385 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:45,385 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:45,385 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:45,385 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:45,387 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,387 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:45,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,387 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:45,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:45,388 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,389 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,391 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:45,392 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:45,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:45,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:45,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,392 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:45,392 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:45,392 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:45,392 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:45,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:45,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397745458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,492 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:45,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:45,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:45,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:45,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:45,496 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:45,496 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:45,496 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:45,496 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:45,497 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,497 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,498 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:45,498 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:45,498 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,498 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:45,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:45,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:45,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397745496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,499 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:45,500 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:45,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,500 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:45,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397745497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,500 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:45,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397745498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,500 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:45,500 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:45,500 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:45,500 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:45,500 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:45,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,500 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:45,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397745498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,500 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:45,500 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,501 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:45,501 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:45,511 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:45,512 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,512 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:45,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:45,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59782098[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:45,513 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:45,513 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,514 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,515 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:45,515 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:45,515 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:45,515 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,515 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:45,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:45,516 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:45,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:45,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:45,519 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:45,519 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:45,519 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:45,519 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:45,519 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:45,520 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,520 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:45,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,520 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:45,521 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,522 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:45,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,526 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:45,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:45,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:45,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:45,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:45,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,527 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,527 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:45,527 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:45,527 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:45,527 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:45,543 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/a65e9f94b48f48f9a6a77956dd9c6d60 2023-04-13 14:54:45,560 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/030a5dce741f4ec08ac8e3a3d844b29a 2023-04-13 14:54:45,585 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/180d16f5e6734e698b57908848a3ab9f 2023-04-13 14:54:45,590 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/a65e9f94b48f48f9a6a77956dd9c6d60 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/a65e9f94b48f48f9a6a77956dd9c6d60 2023-04-13 14:54:45,593 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/a65e9f94b48f48f9a6a77956dd9c6d60, entries=250, sequenceid=68, filesize=16.2 K 2023-04-13 14:54:45,594 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/030a5dce741f4ec08ac8e3a3d844b29a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/030a5dce741f4ec08ac8e3a3d844b29a 2023-04-13 14:54:45,598 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/030a5dce741f4ec08ac8e3a3d844b29a, entries=150, sequenceid=68, filesize=11.6 K 2023-04-13 14:54:45,599 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/180d16f5e6734e698b57908848a3ab9f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/180d16f5e6734e698b57908848a3ab9f 2023-04-13 14:54:45,602 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/180d16f5e6734e698b57908848a3ab9f, entries=150, sequenceid=68, filesize=11.6 K 2023-04-13 14:54:45,603 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for ccd2cd4c855e3a8c5393dd57d5cc737e in 472ms, sequenceid=68, compaction requested=true 2023-04-13 14:54:45,603 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:45,603 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:45,603 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:45,603 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:54:45,603 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:54:45,603 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:45,604 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52224 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:54:45,605 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/A is initiating minor compaction (all files) 2023-04-13 14:54:45,605 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:54:45,605 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/A in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:45,605 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/B is initiating minor compaction (all files) 2023-04-13 14:54:45,605 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d1f0cc8e66954a20ab3a7584f370d796, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/bf1116504a6743d8b60fe9fc5a87bc31, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/769c6646eb76421497b04bde9feaf211, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/a65e9f94b48f48f9a6a77956dd9c6d60] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=51 K 2023-04-13 14:54:45,605 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/B in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:45,605 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/eff61f487c734a628c713b5f82b3ef4e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/94d69f68cf7843948ea1cf87c671f744, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5538dbb055d544f39c7cbfa8212cb68b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/030a5dce741f4ec08ac8e3a3d844b29a] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=46.4 K 2023-04-13 14:54:45,606 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting eff61f487c734a628c713b5f82b3ef4e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681397683838 2023-04-13 14:54:45,606 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting d1f0cc8e66954a20ab3a7584f370d796, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681397683838 2023-04-13 14:54:45,606 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 94d69f68cf7843948ea1cf87c671f744, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681397683874 2023-04-13 14:54:45,606 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting bf1116504a6743d8b60fe9fc5a87bc31, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681397683874 2023-04-13 14:54:45,606 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 5538dbb055d544f39c7cbfa8212cb68b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1681397684490 2023-04-13 14:54:45,606 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 769c6646eb76421497b04bde9feaf211, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1681397684490 2023-04-13 14:54:45,607 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 030a5dce741f4ec08ac8e3a3d844b29a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681397685128 2023-04-13 14:54:45,607 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting a65e9f94b48f48f9a6a77956dd9c6d60, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681397685124 2023-04-13 14:54:45,618 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#B#compaction#194 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:45,626 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#A#compaction#195 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:45,627 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:45,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:45,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:45,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:45,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:45,660 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:45,660 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:45,660 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:45,660 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:45,661 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,661 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,662 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:45,662 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:45,663 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,663 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:45,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:45,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:45,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:45,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:45,671 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:45,671 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:45,672 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-13 14:54:45,672 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:45,672 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:45,672 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:45,672 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:45,672 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:45,672 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:45,674 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/e6349e183f0f447cbe1b4d0a33c69b85 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e6349e183f0f447cbe1b4d0a33c69b85 2023-04-13 14:54:45,680 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/B of ccd2cd4c855e3a8c5393dd57d5cc737e into e6349e183f0f447cbe1b4d0a33c69b85(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:45,680 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:45,680 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/B, priority=12, startTime=1681397685603; duration=0sec 2023-04-13 14:54:45,681 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:45,681 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:54:45,682 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:54:45,682 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/C is initiating minor compaction (all files) 2023-04-13 14:54:45,682 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/C in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:45,682 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/86f19b1950b5419e90280b4a43013e62, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/a39d90b3c7de4bc3ab9b878cb8b04520, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/244f58b4b2b54dedab97704f2427d376, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/180d16f5e6734e698b57908848a3ab9f] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=46.4 K 2023-04-13 14:54:45,683 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 86f19b1950b5419e90280b4a43013e62, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681397683838 2023-04-13 14:54:45,683 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting a39d90b3c7de4bc3ab9b878cb8b04520, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681397683874 2023-04-13 14:54:45,683 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 244f58b4b2b54dedab97704f2427d376, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1681397684490 2023-04-13 14:54:45,684 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 180d16f5e6734e698b57908848a3ab9f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681397685128 2023-04-13 14:54:45,692 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/4d0becfccc8f46258e5bc5482b78fed0 2023-04-13 14:54:45,693 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#C#compaction#197 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:45,702 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/e94f572aeace4497a7cee829339e8460 2023-04-13 14:54:45,714 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/7fdb2dfa1213467fae6eb47f31ed3eb1 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7fdb2dfa1213467fae6eb47f31ed3eb1 2023-04-13 14:54:45,723 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/C of ccd2cd4c855e3a8c5393dd57d5cc737e into 7fdb2dfa1213467fae6eb47f31ed3eb1(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:45,723 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:45,723 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/C, priority=12, startTime=1681397685603; duration=0sec 2023-04-13 14:54:45,723 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:45,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:45,764 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:45,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397745788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397745800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397745802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397745803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397745803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:45,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:45,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397745891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,076 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/e9e190d8dccf42f899f0eb317acf6ead as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/e9e190d8dccf42f899f0eb317acf6ead 2023-04-13 14:54:46,081 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/A of ccd2cd4c855e3a8c5393dd57d5cc737e into e9e190d8dccf42f899f0eb317acf6ead(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:46,081 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 14:54:46,081 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:46,081 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/A, priority=12, startTime=1681397685603; duration=0sec 2023-04-13 14:54:46,081 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:46,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397746093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,130 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/78c04793bd6447638daee15cba692d96 2023-04-13 14:54:46,135 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/4d0becfccc8f46258e5bc5482b78fed0 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/4d0becfccc8f46258e5bc5482b78fed0 2023-04-13 14:54:46,139 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/4d0becfccc8f46258e5bc5482b78fed0, entries=150, sequenceid=104, filesize=11.6 K 2023-04-13 14:54:46,143 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/e94f572aeace4497a7cee829339e8460 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e94f572aeace4497a7cee829339e8460 2023-04-13 14:54:46,147 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e94f572aeace4497a7cee829339e8460, entries=150, sequenceid=104, filesize=11.6 K 2023-04-13 14:54:46,148 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/78c04793bd6447638daee15cba692d96 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/78c04793bd6447638daee15cba692d96 2023-04-13 14:54:46,151 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/78c04793bd6447638daee15cba692d96, entries=150, sequenceid=104, filesize=11.6 K 2023-04-13 14:54:46,152 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for ccd2cd4c855e3a8c5393dd57d5cc737e in 481ms, sequenceid=104, compaction requested=false 2023-04-13 14:54:46,152 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:46,152 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:46,152 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:46,152 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:46,152 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,152 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:46,152 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:46,154 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,154 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:46,154 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,154 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:46,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:46,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43e41450[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:46,156 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:46,156 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,157 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,157 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:46,157 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:46,157 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:46,158 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,158 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:46,160 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,160 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:46,160 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:46,160 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:46,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:46,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:46,163 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:46,163 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:46,163 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:46,163 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:46,163 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:46,164 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,164 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:46,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:46,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,164 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:46,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:46,165 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:46,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:46,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:46,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,167 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:46,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,169 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:46,169 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:46,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:46,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:46,169 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:46,169 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:46,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:46,169 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:46,169 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:46,170 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,170 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,170 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,170 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,170 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:46,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:46,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:46,170 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:46,270 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:46,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:46,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:46,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:46,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:46,274 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:46,274 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:46,274 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:46,274 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:46,274 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,274 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:46,276 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:46,276 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:46,276 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,276 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:46,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:46,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:46,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:46,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:46,277 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:46,277 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:46,277 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:46,278 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:46,278 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:46,278 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:46,278 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:46,278 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:54:46,278 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:46,278 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:46,278 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:46,278 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:46,278 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:46,278 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:46,288 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/c43891b3849143cda93780fb98fd37c3 2023-04-13 14:54:46,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:46,305 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:46,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397686315, max timestamp=1681397686321Num uniques 859; 2023-04-13 14:54:46,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397686315, max timestamp=1681397686321Num uniques 859; 2023-04-13 14:54:46,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397686315, max timestamp=1681397686321Num uniques 859; 2023-04-13 14:54:46,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:46,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:46,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:46,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397746333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397746335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397746336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397746337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397746397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397746437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397746438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397746439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397746440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397746640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397746642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397746642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397746643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,699 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/6cfece40b08042c49bfdb5aef92dcc27 2023-04-13 14:54:46,713 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/3a205fc1bebc417ca344d88fad6c3a16 2023-04-13 14:54:46,719 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/c43891b3849143cda93780fb98fd37c3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/c43891b3849143cda93780fb98fd37c3 2023-04-13 14:54:46,723 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/c43891b3849143cda93780fb98fd37c3, entries=100, sequenceid=117, filesize=9.3 K 2023-04-13 14:54:46,724 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/6cfece40b08042c49bfdb5aef92dcc27 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/6cfece40b08042c49bfdb5aef92dcc27 2023-04-13 14:54:46,730 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/6cfece40b08042c49bfdb5aef92dcc27, entries=100, sequenceid=117, filesize=9.3 K 2023-04-13 14:54:46,731 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/3a205fc1bebc417ca344d88fad6c3a16 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/3a205fc1bebc417ca344d88fad6c3a16 2023-04-13 14:54:46,735 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/3a205fc1bebc417ca344d88fad6c3a16, entries=100, sequenceid=117, filesize=9.3 K 2023-04-13 14:54:46,736 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for ccd2cd4c855e3a8c5393dd57d5cc737e in 458ms, sequenceid=117, compaction requested=true 2023-04-13 14:54:46,736 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:46,736 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:46,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:46,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:46,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:46,737 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:46,741 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,741 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:46,742 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,742 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:46,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:46,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@226f6c0e[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:46,744 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:46,744 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,745 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,745 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:46,745 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:46,745 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:46,746 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,746 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:46,747 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:46,747 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,747 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:46,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:46,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:46,747 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:46,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:46,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:46,750 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:46,751 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:46,751 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:46,751 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:46,751 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:46,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:46,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:46,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,754 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,755 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,758 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,758 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:46,759 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:46,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:46,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:46,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:46,758 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:46,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:46,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,759 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:46,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:46,759 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:46,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:46,859 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:46,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:46,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:46,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:46,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:46,863 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:46,863 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:46,863 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:46,863 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:46,864 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,864 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:46,865 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:46,865 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:46,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:46,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:46,866 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,866 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:46,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:46,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:46,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:46,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:46,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:46,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:46,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:46,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:46,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:46,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:46,867 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:46,867 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:46,867 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-13 14:54:46,868 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:46,868 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:46,868 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:46,868 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:46,868 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:46,868 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:46,879 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/237608c95e0740d49614d0979d62ecbb 2023-04-13 14:54:46,891 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/e497e49a2dd0482d85b6a8b768361d23 2023-04-13 14:54:46,903 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/6b0a16bae25c4ebf901ea1aabff163e7 2023-04-13 14:54:46,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:46,904 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:46,908 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/237608c95e0740d49614d0979d62ecbb as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/237608c95e0740d49614d0979d62ecbb 2023-04-13 14:54:46,912 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/237608c95e0740d49614d0979d62ecbb, entries=150, sequenceid=154, filesize=11.8 K 2023-04-13 14:54:46,912 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/e497e49a2dd0482d85b6a8b768361d23 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e497e49a2dd0482d85b6a8b768361d23 2023-04-13 14:54:46,916 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e497e49a2dd0482d85b6a8b768361d23, entries=150, sequenceid=154, filesize=11.8 K 2023-04-13 14:54:46,917 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/6b0a16bae25c4ebf901ea1aabff163e7 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/6b0a16bae25c4ebf901ea1aabff163e7 2023-04-13 14:54:46,930 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/6b0a16bae25c4ebf901ea1aabff163e7, entries=150, sequenceid=154, filesize=11.8 K 2023-04-13 14:54:46,931 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=33.54 KB/34350 for ccd2cd4c855e3a8c5393dd57d5cc737e in 64ms, sequenceid=154, compaction requested=true 2023-04-13 14:54:46,931 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:46,932 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:46,932 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:46,932 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:46,932 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,932 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:46,932 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:46,933 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,933 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:46,934 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,934 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:46,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:46,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@324c2366[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:46,937 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:46,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:46,937 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,937 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:54:46,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:46,937 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:46,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:46,937 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:46,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:46,937 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:46,942 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,943 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:46,943 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:46,943 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:46,943 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,943 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:46,945 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:46,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:46,949 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:46,949 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:46,949 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:46,949 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:46,949 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:46,951 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,951 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:46,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,951 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:46,952 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681397686952, max timestamp=1681397686954Num uniques 859; 2023-04-13 14:54:46,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:46,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681397686952, max timestamp=1681397686954Num uniques 859; 2023-04-13 14:54:46,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:46,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397686952, max timestamp=1681397686956Num uniques 859; 2023-04-13 14:54:46,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,960 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,962 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:46,962 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,962 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:46,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:46,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:46,962 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:46,962 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:46,962 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:46,963 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:46,963 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:46,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:46,963 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,963 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,963 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:46,963 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:46,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:46,979 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/c264d6b9a9694fe9a125f8df52d029e3 2023-04-13 14:54:46,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397746974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397746980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397746979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397746983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:46,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:46,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397746985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,063 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:47,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:47,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:47,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:47,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:47,067 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:47,067 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:47,067 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:47,067 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:47,068 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,068 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,070 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:47,070 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:47,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:47,070 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,070 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:47,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:47,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:47,072 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:47,072 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:47,072 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:47,072 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:47,073 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:47,073 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:47,073 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:47,073 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:47,073 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:47,073 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:47,073 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:47,073 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,073 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:47,073 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:47,074 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:47,075 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,075 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,075 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:47,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:47,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:47,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:47,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a28b957[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:47,077 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:47,077 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,079 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,079 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:47,079 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:47,079 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,079 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:47,079 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:47,081 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:47,081 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:47,081 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:47,081 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:47,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:47,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:47,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:47,084 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:47,084 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:47,084 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:47,084 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:47,084 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:47,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397747085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,090 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397747088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:47,090 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,090 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:47,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:47,090 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397747089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,092 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397747089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:47,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397747089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,094 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,094 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,094 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,094 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:47,094 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:47,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:47,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:47,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:47,094 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:47,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:47,094 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:47,094 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:47,094 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:47,094 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,095 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,095 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:47,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:47,195 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:47,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:47,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:47,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:47,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:47,198 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:47,198 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:47,198 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:47,198 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:47,198 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,198 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,200 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:47,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:47,200 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:47,200 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,200 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:47,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:47,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:47,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:47,201 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:47,201 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:47,201 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:47,201 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:47,202 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:47,202 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:47,202 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:47,202 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:47,202 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:47,202 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:47,202 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:47,202 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,202 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:47,202 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:47,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,204 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:47,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:47,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:47,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:47,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@490d6ed6[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:47,209 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:47,209 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,211 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,211 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:47,211 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,212 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:47,212 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:47,212 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:47,213 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,213 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:47,213 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:47,213 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:47,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:47,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:47,216 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:47,216 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:47,216 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:47,216 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:47,216 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:47,218 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,218 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:47,219 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:47,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:47,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:47,220 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,221 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,223 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,223 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,224 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,224 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,224 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,224 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,224 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,224 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:47,224 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:47,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:47,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:47,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:47,224 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:47,224 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:47,224 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:47,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:47,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:47,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397747289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397747291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397747293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397747293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397747294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,324 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:47,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:47,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:47,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:47,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:47,328 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:47,328 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:47,328 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:47,328 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:47,328 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,328 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,330 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,330 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:47,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:47,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:47,331 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,331 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:47,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:47,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:47,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:47,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:47,332 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:47,332 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:47,332 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:47,333 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:47,333 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:47,333 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:47,333 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:47,335 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:47,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:47,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10db74e5[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:47,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:47,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:47,339 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:47,339 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,339 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:47,344 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:47,344 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:47,344 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,344 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:47,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:47,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:47,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:47,346 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:47,346 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:47,346 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:47,346 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:47,346 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:47,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:47,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:47,348 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,349 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,353 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:47,353 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:47,353 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:47,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:47,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:47,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:47,354 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:47,353 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,354 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,354 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,354 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:47,354 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:47,354 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,354 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,354 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:47,402 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/805178f4cbcc480da7ed467314029570 2023-04-13 14:54:47,414 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/c3dc619531dc4f8294877d45c4a9c274 2023-04-13 14:54:47,418 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/c264d6b9a9694fe9a125f8df52d029e3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/c264d6b9a9694fe9a125f8df52d029e3 2023-04-13 14:54:47,422 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/c264d6b9a9694fe9a125f8df52d029e3, entries=150, sequenceid=164, filesize=11.8 K 2023-04-13 14:54:47,422 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/805178f4cbcc480da7ed467314029570 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/805178f4cbcc480da7ed467314029570 2023-04-13 14:54:47,428 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/805178f4cbcc480da7ed467314029570, entries=100, sequenceid=164, filesize=9.4 K 2023-04-13 14:54:47,429 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/c3dc619531dc4f8294877d45c4a9c274 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/c3dc619531dc4f8294877d45c4a9c274 2023-04-13 14:54:47,434 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/c3dc619531dc4f8294877d45c4a9c274, entries=100, sequenceid=164, filesize=9.4 K 2023-04-13 14:54:47,435 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ccd2cd4c855e3a8c5393dd57d5cc737e in 498ms, sequenceid=164, compaction requested=true 2023-04-13 14:54:47,435 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:47,435 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:47,435 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 14:54:47,435 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 14:54:47,435 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:47,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:47,439 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57520 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 14:54:47,439 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/A is initiating minor compaction (all files) 2023-04-13 14:54:47,439 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55130 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 14:54:47,439 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/A in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:47,440 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/B is initiating minor compaction (all files) 2023-04-13 14:54:47,440 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/e9e190d8dccf42f899f0eb317acf6ead, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/4d0becfccc8f46258e5bc5482b78fed0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/c43891b3849143cda93780fb98fd37c3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/237608c95e0740d49614d0979d62ecbb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/c264d6b9a9694fe9a125f8df52d029e3] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=56.2 K 2023-04-13 14:54:47,440 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/B in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:47,440 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e6349e183f0f447cbe1b4d0a33c69b85, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e94f572aeace4497a7cee829339e8460, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/6cfece40b08042c49bfdb5aef92dcc27, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e497e49a2dd0482d85b6a8b768361d23, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/805178f4cbcc480da7ed467314029570] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=53.8 K 2023-04-13 14:54:47,440 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting e9e190d8dccf42f899f0eb317acf6ead, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681397685128 2023-04-13 14:54:47,440 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting e6349e183f0f447cbe1b4d0a33c69b85, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681397685128 2023-04-13 14:54:47,440 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 4d0becfccc8f46258e5bc5482b78fed0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681397685183 2023-04-13 14:54:47,440 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting e94f572aeace4497a7cee829339e8460, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681397685183 2023-04-13 14:54:47,441 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting c43891b3849143cda93780fb98fd37c3, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681397685777 2023-04-13 14:54:47,441 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 6cfece40b08042c49bfdb5aef92dcc27, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681397685777 2023-04-13 14:54:47,441 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 237608c95e0740d49614d0979d62ecbb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681397686332 2023-04-13 14:54:47,441 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting e497e49a2dd0482d85b6a8b768361d23, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681397686332 2023-04-13 14:54:47,442 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting c264d6b9a9694fe9a125f8df52d029e3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1681397686930 2023-04-13 14:54:47,442 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 805178f4cbcc480da7ed467314029570, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1681397686930 2023-04-13 14:54:47,450 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#B#compaction#209 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:47,451 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#A#compaction#210 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:47,454 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:47,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:47,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:47,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:47,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:47,466 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:47,466 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:47,466 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:47,466 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:47,467 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,467 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,468 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,468 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:47,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,472 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,472 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:47,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:47,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:47,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:47,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:47,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:47,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:47,488 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:47,488 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:47,488 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:47,492 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:47,492 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:47,492 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:47,492 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:47,492 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-13 14:54:47,492 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:47,492 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:47,493 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:47,493 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:47,493 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:47,493 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:47,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:47,593 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:47,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397747599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397747599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397747599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397747601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397747601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397747703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397747703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397747706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397747706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,876 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/0f65d0feed0f4f7a94beaff1878c50a0 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/0f65d0feed0f4f7a94beaff1878c50a0 2023-04-13 14:54:47,880 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/A of ccd2cd4c855e3a8c5393dd57d5cc737e into 0f65d0feed0f4f7a94beaff1878c50a0(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:47,881 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:47,881 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/A, priority=11, startTime=1681397687435; duration=0sec 2023-04-13 14:54:47,881 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:47,881 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 14:54:47,882 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55130 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 14:54:47,883 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/C is initiating minor compaction (all files) 2023-04-13 14:54:47,883 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/C in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:47,883 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7fdb2dfa1213467fae6eb47f31ed3eb1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/78c04793bd6447638daee15cba692d96, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/3a205fc1bebc417ca344d88fad6c3a16, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/6b0a16bae25c4ebf901ea1aabff163e7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/c3dc619531dc4f8294877d45c4a9c274] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=53.8 K 2023-04-13 14:54:47,883 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 7fdb2dfa1213467fae6eb47f31ed3eb1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681397685128 2023-04-13 14:54:47,884 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 78c04793bd6447638daee15cba692d96, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681397685183 2023-04-13 14:54:47,884 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 3a205fc1bebc417ca344d88fad6c3a16, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681397685777 2023-04-13 14:54:47,884 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 6b0a16bae25c4ebf901ea1aabff163e7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681397686332 2023-04-13 14:54:47,885 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting c3dc619531dc4f8294877d45c4a9c274, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1681397686930 2023-04-13 14:54:47,891 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/c2a208edfb394f388c63ff5f6aaa58a6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/c2a208edfb394f388c63ff5f6aaa58a6 2023-04-13 14:54:47,896 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#C#compaction#212 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:47,896 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/B of ccd2cd4c855e3a8c5393dd57d5cc737e into c2a208edfb394f388c63ff5f6aaa58a6(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:47,896 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:47,896 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/B, priority=11, startTime=1681397687435; duration=0sec 2023-04-13 14:54:47,897 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:47,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397747908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397747908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397747908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,914 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/d22c14771ac64ab7a10d8020be661642 2023-04-13 14:54:47,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:47,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397747911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,929 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/201f01cbc2534bbdb3e3f34c1dac2681 2023-04-13 14:54:47,949 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/57dc49729faa450bb8d37e9b31cc81bd 2023-04-13 14:54:47,954 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/d22c14771ac64ab7a10d8020be661642 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d22c14771ac64ab7a10d8020be661642 2023-04-13 14:54:47,958 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d22c14771ac64ab7a10d8020be661642, entries=150, sequenceid=200, filesize=11.8 K 2023-04-13 14:54:47,959 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/201f01cbc2534bbdb3e3f34c1dac2681 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/201f01cbc2534bbdb3e3f34c1dac2681 2023-04-13 14:54:47,980 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/201f01cbc2534bbdb3e3f34c1dac2681, entries=150, sequenceid=200, filesize=11.8 K 2023-04-13 14:54:47,983 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/57dc49729faa450bb8d37e9b31cc81bd as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/57dc49729faa450bb8d37e9b31cc81bd 2023-04-13 14:54:47,990 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/57dc49729faa450bb8d37e9b31cc81bd, entries=150, sequenceid=200, filesize=11.8 K 2023-04-13 14:54:47,991 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for ccd2cd4c855e3a8c5393dd57d5cc737e in 499ms, sequenceid=200, compaction requested=false 2023-04-13 14:54:47,991 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:47,991 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:47,994 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,994 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:47,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:47,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:47,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43df228b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:47,997 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:47,997 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,998 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:47,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:47,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:47,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:47,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:47,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:48,000 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:48,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:48,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:48,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:48,003 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:48,003 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:48,003 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:48,003 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:48,003 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:48,005 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,005 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,005 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:48,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:48,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:48,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,007 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:48,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:48,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:48,012 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:48,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:48,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:48,012 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,012 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,012 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,012 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,012 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,012 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:48,012 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:48,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:48,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:48,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:48,108 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 14:54:48,108 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:48,108 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:48,108 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:48,108 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:48,108 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:48,108 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:48,112 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:48,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:48,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:48,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:48,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:48,119 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:48,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:48,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:48,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:48,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,120 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:48,120 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,121 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,121 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:48,121 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/ca2103d9d837486f949e062b5e1a3c83 2023-04-13 14:54:48,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:48,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:48,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:48,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:48,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:48,123 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:48,123 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:48,123 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:48,123 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:48,125 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,125 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:48,125 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:48,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:48,127 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:48,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c52d9a6[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:48,127 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,129 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:48,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:48,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:48,129 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,129 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:48,131 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,131 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:48,131 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:48,131 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:48,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:48,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:48,134 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:48,134 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:48,134 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:48,134 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:48,135 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:48,136 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/78999fa6d38d4c9a99cae62da7f03519 2023-04-13 14:54:48,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:48,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:48,138 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:48,139 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,142 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,142 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,142 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,142 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:48,142 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,142 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,142 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,142 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,142 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:48,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:48,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:48,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:48,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:48,142 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:48,142 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:48,142 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:48,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:48,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:48,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397688154, max timestamp=1681397688170Num uniques 859; 2023-04-13 14:54:48,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:48,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397688154, max timestamp=1681397688170Num uniques 859; 2023-04-13 14:54:48,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:48,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397688154, max timestamp=1681397688170Num uniques 859; 2023-04-13 14:54:48,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:48,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397748214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397748215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397748216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397748217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397748218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,242 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:48,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:48,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:48,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:48,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:48,246 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:48,246 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:48,246 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:48,246 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:48,247 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,247 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,248 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:48,248 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:48,248 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,248 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:48,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:48,249 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:48,249 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:48,250 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:48,252 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:48,252 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:48,252 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:48,252 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:48,255 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:48,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:48,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:48,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e0b6924[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:48,256 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:48,257 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,258 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,258 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:48,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:48,258 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:48,258 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:48,260 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:48,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:48,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:48,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:48,262 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:48,262 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:48,262 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:48,262 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:48,262 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:48,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:48,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:48,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:48,264 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,265 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,271 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,272 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:48,272 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,272 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:48,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:48,272 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:48,272 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:48,272 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:48,272 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,272 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:48,272 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,272 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:48,272 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:48,307 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/c7ef38638c4040308120e967e3d994c2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/c7ef38638c4040308120e967e3d994c2 2023-04-13 14:54:48,311 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/C of ccd2cd4c855e3a8c5393dd57d5cc737e into c7ef38638c4040308120e967e3d994c2(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:48,311 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:48,311 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/C, priority=11, startTime=1681397687436; duration=0sec 2023-04-13 14:54:48,311 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:48,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397748319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397748321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397748324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397748324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,372 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:48,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:48,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:48,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:48,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:48,376 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:48,376 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:48,376 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:48,376 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:48,377 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,377 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,378 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:48,378 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:48,378 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,378 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:48,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:48,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:48,380 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:48,380 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:48,380 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:48,380 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:48,382 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,382 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:48,382 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,382 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453c585c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:48,384 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:48,384 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,385 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:48,385 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,385 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:48,387 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:48,388 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:48,388 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:48,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:48,390 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:48,390 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:48,390 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:48,390 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:48,390 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:48,392 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:48,392 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,392 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:48,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:48,392 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:48,393 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,395 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:48,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:48,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:48,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:48,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:48,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:48,396 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:48,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:48,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:48,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:48,396 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:48,396 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:48,497 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:48,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:48,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:48,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:48,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:48,500 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:48,500 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:48,500 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:48,500 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:48,501 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,501 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,502 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:48,502 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:48,502 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,502 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:48,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:48,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:48,503 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:48,503 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:48,503 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:48,504 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:48,504 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:48,504 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:48,504 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:48,506 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,506 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:48,507 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,507 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:48,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:48,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fcdf10a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:48,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:48,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,509 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,510 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:48,510 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:48,510 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:48,510 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,510 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:48,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:48,511 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:48,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:48,513 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:48,513 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:48,513 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:48,513 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:48,513 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:48,515 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,515 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:48,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,515 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:48,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,515 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,517 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,520 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:48,520 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:48,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:48,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:48,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:48,520 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:48,520 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,520 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,520 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,520 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,520 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:48,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:48,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:48,520 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:48,520 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,520 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:48,520 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397748523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397748525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397748527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397748527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,553 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/d1f3c505ee2f4ed0addbad3ac6e6e322 2023-04-13 14:54:48,557 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/ca2103d9d837486f949e062b5e1a3c83 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/ca2103d9d837486f949e062b5e1a3c83 2023-04-13 14:54:48,561 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/ca2103d9d837486f949e062b5e1a3c83, entries=200, sequenceid=213, filesize=14.1 K 2023-04-13 14:54:48,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/78999fa6d38d4c9a99cae62da7f03519 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/78999fa6d38d4c9a99cae62da7f03519 2023-04-13 14:54:48,565 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/78999fa6d38d4c9a99cae62da7f03519, entries=150, sequenceid=213, filesize=11.8 K 2023-04-13 14:54:48,566 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/d1f3c505ee2f4ed0addbad3ac6e6e322 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/d1f3c505ee2f4ed0addbad3ac6e6e322 2023-04-13 14:54:48,570 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/d1f3c505ee2f4ed0addbad3ac6e6e322, entries=150, sequenceid=213, filesize=11.8 K 2023-04-13 14:54:48,570 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for ccd2cd4c855e3a8c5393dd57d5cc737e in 462ms, sequenceid=213, compaction requested=true 2023-04-13 14:54:48,571 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:48,571 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:48,571 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:48,571 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:48,571 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:48,571 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:48,572 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38806 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:48,572 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/A is initiating minor compaction (all files) 2023-04-13 14:54:48,572 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:48,572 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/A in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:48,572 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/B is initiating minor compaction (all files) 2023-04-13 14:54:48,572 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/0f65d0feed0f4f7a94beaff1878c50a0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d22c14771ac64ab7a10d8020be661642, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/ca2103d9d837486f949e062b5e1a3c83] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=37.9 K 2023-04-13 14:54:48,572 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/B in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:48,572 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/c2a208edfb394f388c63ff5f6aaa58a6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/201f01cbc2534bbdb3e3f34c1dac2681, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/78999fa6d38d4c9a99cae62da7f03519] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=35.6 K 2023-04-13 14:54:48,572 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 0f65d0feed0f4f7a94beaff1878c50a0, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1681397686332 2023-04-13 14:54:48,573 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting c2a208edfb394f388c63ff5f6aaa58a6, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1681397686332 2023-04-13 14:54:48,573 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting d22c14771ac64ab7a10d8020be661642, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1681397686968 2023-04-13 14:54:48,573 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 201f01cbc2534bbdb3e3f34c1dac2681, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1681397686968 2023-04-13 14:54:48,573 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting ca2103d9d837486f949e062b5e1a3c83, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681397687598 2023-04-13 14:54:48,573 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 78999fa6d38d4c9a99cae62da7f03519, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681397687600 2023-04-13 14:54:48,579 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#B#compaction#219 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:48,580 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#A#compaction#218 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:48,620 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:48,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:48,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:48,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:48,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:48,624 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:48,624 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:48,624 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:48,624 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:48,624 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,624 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,625 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:48,625 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:48,626 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,626 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:48,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:48,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:48,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:48,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:48,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:48,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:48,627 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:48,627 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:48,627 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:48,628 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:48,628 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:48,628 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:48,628 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:48,628 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-13 14:54:48,628 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:48,628 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:48,628 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:48,629 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:48,629 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:48,629 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:48,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:48,728 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:48,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397748750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397748828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397748831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397748831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397748832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:48,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397748852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:48,993 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/791ffc946ee54a4c879fda2340fdfdc3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/791ffc946ee54a4c879fda2340fdfdc3 2023-04-13 14:54:48,997 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/81fbcbc04bf44564b9579f7f0fb861b0 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/81fbcbc04bf44564b9579f7f0fb861b0 2023-04-13 14:54:48,999 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/B of ccd2cd4c855e3a8c5393dd57d5cc737e into 791ffc946ee54a4c879fda2340fdfdc3(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:48,999 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:48,999 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/B, priority=13, startTime=1681397688571; duration=0sec 2023-04-13 14:54:48,999 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:48,999 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:49,001 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:49,001 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/C is initiating minor compaction (all files) 2023-04-13 14:54:49,001 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/C in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:49,001 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/c7ef38638c4040308120e967e3d994c2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/57dc49729faa450bb8d37e9b31cc81bd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/d1f3c505ee2f4ed0addbad3ac6e6e322] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=35.6 K 2023-04-13 14:54:49,002 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting c7ef38638c4040308120e967e3d994c2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1681397686332 2023-04-13 14:54:49,004 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/A of ccd2cd4c855e3a8c5393dd57d5cc737e into 81fbcbc04bf44564b9579f7f0fb861b0(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:49,004 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:49,004 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 57dc49729faa450bb8d37e9b31cc81bd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1681397686968 2023-04-13 14:54:49,004 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/A, priority=13, startTime=1681397688571; duration=0sec 2023-04-13 14:54:49,004 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:49,004 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting d1f3c505ee2f4ed0addbad3ac6e6e322, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681397687600 2023-04-13 14:54:49,011 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#C#compaction#221 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:49,035 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/58b9f59613fc4e67bd90b6f1ea50190d 2023-04-13 14:54:49,054 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/04969316840145adbd733b32504b8841 2023-04-13 14:54:49,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:49,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397749054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:49,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397749334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:49,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397749337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:49,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397749338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:49,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397749338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:49,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53214 deadline: 1681397749356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,435 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/798053fae5a84e88ab0904b32057b12f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/798053fae5a84e88ab0904b32057b12f 2023-04-13 14:54:49,439 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/C of ccd2cd4c855e3a8c5393dd57d5cc737e into 798053fae5a84e88ab0904b32057b12f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:49,439 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:49,439 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/C, priority=13, startTime=1681397688571; duration=0sec 2023-04-13 14:54:49,439 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:49,466 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/f94a511f13724a54af13b996a87ca11a 2023-04-13 14:54:49,470 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/58b9f59613fc4e67bd90b6f1ea50190d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/58b9f59613fc4e67bd90b6f1ea50190d 2023-04-13 14:54:49,474 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/58b9f59613fc4e67bd90b6f1ea50190d, entries=150, sequenceid=250, filesize=11.8 K 2023-04-13 14:54:49,475 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/04969316840145adbd733b32504b8841 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/04969316840145adbd733b32504b8841 2023-04-13 14:54:49,479 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/04969316840145adbd733b32504b8841, entries=150, sequenceid=250, filesize=11.8 K 2023-04-13 14:54:49,480 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/f94a511f13724a54af13b996a87ca11a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/f94a511f13724a54af13b996a87ca11a 2023-04-13 14:54:49,483 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/f94a511f13724a54af13b996a87ca11a, entries=150, sequenceid=250, filesize=11.8 K 2023-04-13 14:54:49,484 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for ccd2cd4c855e3a8c5393dd57d5cc737e in 856ms, sequenceid=250, compaction requested=false 2023-04-13 14:54:49,484 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:49,484 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:49,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:49,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:49,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:49,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:49,486 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,486 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:49,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:49,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:49,486 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:49,487 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:49,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:49,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@456898bc[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:49,488 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:49,488 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:49,490 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:49,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:49,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:49,490 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:49,490 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:49,490 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,490 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:49,490 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:49,492 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,492 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:49,492 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:49,492 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:49,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:49,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:49,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:49,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:49,495 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:49,495 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:49,495 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:49,495 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:49,495 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:49,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:49,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:49,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:49,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:49,498 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:49,499 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,504 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,504 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:49,504 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:49,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:49,504 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:49,505 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:49,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:49,504 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:49,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:49,505 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:49,505 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:49,505 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:49,505 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:49,505 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:49,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:49,505 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:49,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:49,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:49,605 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:49,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:49,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:49,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:49,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:49,609 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:49,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:49,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:49,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:49,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:49,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:49,610 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,610 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:49,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:49,611 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:49,611 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:49,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:49,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:49,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:49,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:49,612 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:49,612 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:49,612 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:49,612 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:49,612 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:49,612 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:49,612 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:49,612 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:54:49,612 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:49,612 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:49,613 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:49,613 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:49,613 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:49,613 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:49,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:49,864 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:49,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397689905, max timestamp=1681397689919Num uniques 859; 2023-04-13 14:54:49,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:54:49,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397689905, max timestamp=1681397689919Num uniques 859; 2023-04-13 14:54:49,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:54:49,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397689905, max timestamp=1681397689919Num uniques 859; 2023-04-13 14:54:49,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:54:49,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:49,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53214 deadline: 1681397749976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,021 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/598577a1df2e495c84460c16a3420717 2023-04-13 14:54:50,031 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/835af86082e84722bbeae714bbba51fa 2023-04-13 14:54:50,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:50,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53214 deadline: 1681397750079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:50,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53214 deadline: 1681397750283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:50,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:50,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397750342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397750342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:50,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397750345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:50,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397750346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,444 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/42040992c6814e23bd6b19cd534f23f9 2023-04-13 14:54:50,453 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/598577a1df2e495c84460c16a3420717 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/598577a1df2e495c84460c16a3420717 2023-04-13 14:54:50,458 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/598577a1df2e495c84460c16a3420717, entries=150, sequenceid=263, filesize=11.9 K 2023-04-13 14:54:50,459 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/835af86082e84722bbeae714bbba51fa as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/835af86082e84722bbeae714bbba51fa 2023-04-13 14:54:50,474 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/835af86082e84722bbeae714bbba51fa, entries=150, sequenceid=263, filesize=11.9 K 2023-04-13 14:54:50,474 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/42040992c6814e23bd6b19cd534f23f9 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/42040992c6814e23bd6b19cd534f23f9 2023-04-13 14:54:50,478 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/42040992c6814e23bd6b19cd534f23f9, entries=150, sequenceid=263, filesize=11.9 K 2023-04-13 14:54:50,478 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ccd2cd4c855e3a8c5393dd57d5cc737e in 866ms, sequenceid=263, compaction requested=true 2023-04-13 14:54:50,479 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:50,479 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:50,479 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:50,479 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:50,479 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,479 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:50,479 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:50,481 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:50,481 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:50,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:50,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:50,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ff8b7c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:50,483 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:50,483 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,485 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,485 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:50,485 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:50,485 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:50,485 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,485 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:50,486 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,486 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:50,486 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:50,487 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:50,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:50,489 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:50,489 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:50,489 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:50,489 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:50,489 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:50,491 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,491 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,491 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:50,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:50,492 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,493 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,496 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,496 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,496 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,496 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,496 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:50,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:50,497 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:50,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:50,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:50,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:50,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:50,497 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:50,497 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:50,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:50,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:50,592 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-13 14:54:50,592 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:50,592 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:54:50,592 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:50,592 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:54:50,592 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:50,592 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:54:50,597 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:50,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:50,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:50,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:50,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:50,611 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:50,611 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:50,611 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:50,611 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:50,613 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,613 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,615 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,615 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:50,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:50,615 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,615 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:50,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:50,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:50,617 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:50,617 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:50,617 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:50,617 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,619 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:50,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:50,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3726cf87[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:50,622 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:50,622 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,623 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:50,624 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,624 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:50,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:50,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53214 deadline: 1681397750623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,626 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:50,626 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:50,626 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:50,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:50,630 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:50,630 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:50,630 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:50,630 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:50,630 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:50,631 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,631 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,632 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:50,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:50,632 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:50,633 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:50,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:50,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:50,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:50,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:50,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:50,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:50,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:50,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,637 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:50,637 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:50,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:50,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53214 deadline: 1681397750726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,737 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:50,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:50,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:50,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:50,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:50,740 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:50,740 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:50,740 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:50,740 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:50,740 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,740 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,741 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:50,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:50,741 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:50,742 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,742 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:50,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:50,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:50,743 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:50,743 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:50,743 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:50,743 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:50,743 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:50,743 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:50,743 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:50,743 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:50,743 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:50,743 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:50,743 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:50,743 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,744 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:50,744 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:50,746 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,746 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:50,746 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,746 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:50,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:50,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27f32dc8[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:50,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:50,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,749 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,749 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:50,749 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:50,749 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:50,749 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,750 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:50,750 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:50,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:50,753 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:50,753 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:50,753 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:50,753 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:50,753 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:50,755 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,755 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,755 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:50,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,755 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:50,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,756 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,760 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:50,760 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,760 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:50,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:50,760 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:50,760 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,760 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,760 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,760 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,761 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,761 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:50,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:50,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:50,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:50,761 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:50,761 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:50,861 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:50,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:50,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:50,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:50,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:50,866 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:50,866 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:50,866 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:50,867 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:50,867 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,867 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,871 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,871 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:50,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,872 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,872 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:50,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:50,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:50,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:50,873 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:50,873 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:50,873 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:50,873 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:50,875 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,875 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:50,875 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,875 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:50,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:50,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26dfecf2[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:50,877 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:50,877 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,879 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,879 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:50,879 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:50,879 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:50,879 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,879 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:50,880 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,880 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:50,881 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:50,881 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:50,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:50,883 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:50,883 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:50,883 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:50,883 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:50,883 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:50,885 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,885 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:50,885 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:50,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:50,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,885 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:50,887 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,889 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,889 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:50,889 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:50,889 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:50,889 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,889 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,889 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,889 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:50,889 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:50,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:50,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:50,890 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:50,890 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:50,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:50,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53214 deadline: 1681397750931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,990 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:50,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:50,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:50,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:50,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:50,993 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:50,993 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:50,993 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:50,993 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:50,994 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,994 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,996 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:50,996 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:50,996 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,996 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:50,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:50,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:50,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:50,998 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:50,998 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:50,998 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:50,998 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:50,999 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:50,999 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:51,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a97273e[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:51,001 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:51,001 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:51,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:51,003 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,003 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:51,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:51,004 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,004 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:51,004 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,004 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:51,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:51,006 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:51,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:51,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:51,007 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:51,007 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:51,009 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,009 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,009 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,009 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,010 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,013 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,013 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,013 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,013 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,013 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:51,013 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:51,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:51,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:51,013 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:51,013 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,013 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:51,013 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:51,013 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:51,013 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:51,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:51,015 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/270616b5d5ba40e19a8764472bce39a9 2023-04-13 14:54:51,113 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:51,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:51,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:51,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:51,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:51,117 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:51,117 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:51,117 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:51,117 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:51,117 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,117 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,118 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:51,118 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:51,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:51,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:51,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:51,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:51,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:51,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:51,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:51,120 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,120 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:51,120 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:51,120 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:51,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,123 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,124 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,124 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:51,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:51,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bcc931c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:51,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,127 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,127 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:51,127 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:51,127 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:51,127 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,127 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:51,129 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:51,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:51,131 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:51,131 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:51,131 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:51,131 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:51,131 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:51,133 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,133 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,133 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:51,133 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,135 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:51,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:51,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:51,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:51,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:51,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:51,137 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:51,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,137 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:51,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:51,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:51,137 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:51,238 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:51,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:51,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53214 deadline: 1681397751236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:51,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:51,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:51,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:51,240 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:51,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:51,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:51,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:51,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,244 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:51,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:51,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:51,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:51,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:51,245 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:51,245 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:51,245 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:51,245 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:51,245 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:51,245 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,246 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:51,246 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:51,246 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:51,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,247 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,247 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,247 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:51,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@756f7ac0[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:51,249 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:51,249 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,252 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:51,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:51,252 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:51,253 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:51,255 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:51,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:51,258 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:51,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:51,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:51,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:51,258 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:51,259 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,259 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,259 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,260 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,261 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:51,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:51,264 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:51,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:51,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:51,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:51,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:51,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:51,265 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:51,265 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:51,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:51,365 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:51,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:51,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:51,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:51,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:51,387 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:51,387 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:51,388 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:51,388 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:51,388 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,388 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,390 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:51,390 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:51,390 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,390 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:51,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:51,392 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,392 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:51,392 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:51,392 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:51,394 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,394 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,394 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,395 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:51,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c0ea0e3[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:51,396 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:51,396 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,397 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,398 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:51,398 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:51,398 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:51,398 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,398 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:51,399 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,399 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:51,399 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:51,399 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:51,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:51,401 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:51,401 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:51,401 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:51,401 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:51,401 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:51,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:51,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,404 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,405 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,408 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,408 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:51,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:51,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:51,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:51,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:51,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,409 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:51,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:51,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,409 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:51,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,409 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:51,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:51,426 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/70ceef37130943959f8f83453aeeac8e 2023-04-13 14:54:51,509 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:51,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:51,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:51,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:51,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:51,513 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:51,513 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:51,513 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:51,513 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:51,513 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,513 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,515 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,515 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:51,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:51,515 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,515 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:51,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:51,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:51,516 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,517 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:51,517 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:51,517 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,517 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:51,517 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:51,517 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,517 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:51,517 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:51,518 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,518 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,519 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,519 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:51,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fd3dc8b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:51,520 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:51,520 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,521 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:51,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:51,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:51,522 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,522 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:51,523 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,524 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:51,524 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:51,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:51,526 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:51,526 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:51,526 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:51,526 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:51,526 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:51,528 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,528 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,528 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:51,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,529 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:51,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:51,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:51,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:51,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:51,532 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:51,533 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:51,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:51,533 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:51,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:51,633 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:51,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:51,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:51,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:51,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:51,636 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:51,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:51,637 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:51,637 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:51,637 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,637 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,639 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,639 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:51,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:51,639 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,639 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:51,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:51,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:51,640 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:51,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:51,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:51,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:51,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:51,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7028f01e[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:51,645 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:51,645 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,646 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:51,647 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:51,647 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:51,647 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,647 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:51,651 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:51,653 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:51,653 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:51,654 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:51,654 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:51,654 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:51,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,656 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,657 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,660 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:51,660 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:51,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:51,660 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:51,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:51,660 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:51,660 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:51,660 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,661 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:51,661 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:51,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:51,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:51,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53214 deadline: 1681397751743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,761 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:51,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:51,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:51,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:51,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:51,764 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:51,764 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:51,764 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:51,764 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:51,765 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,765 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,767 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:51,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:51,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:51,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:51,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:51,769 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,769 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:51,769 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:51,769 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,771 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:51,773 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:51,773 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e1f2da1[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:51,775 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,775 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:51,775 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:51,775 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:51,775 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,776 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:51,777 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,777 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:51,777 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,777 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:51,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:51,781 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:51,781 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:51,781 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:51,781 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:51,781 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:51,782 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:51,782 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,783 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,783 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:51,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:51,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,784 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,786 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:51,786 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,786 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:51,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:51,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:51,786 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,786 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:51,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:51,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:51,786 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:51,786 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,786 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,786 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:51,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,786 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:51,787 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:51,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:51,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:51,840 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/bf33403a916044e9a2f3086ac6b3ebb1 2023-04-13 14:54:51,845 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/270616b5d5ba40e19a8764472bce39a9 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/270616b5d5ba40e19a8764472bce39a9 2023-04-13 14:54:51,849 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/270616b5d5ba40e19a8764472bce39a9, entries=200, sequenceid=301, filesize=14.3 K 2023-04-13 14:54:51,849 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/70ceef37130943959f8f83453aeeac8e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/70ceef37130943959f8f83453aeeac8e 2023-04-13 14:54:51,854 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/70ceef37130943959f8f83453aeeac8e, entries=150, sequenceid=301, filesize=11.9 K 2023-04-13 14:54:51,854 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/bf33403a916044e9a2f3086ac6b3ebb1 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/bf33403a916044e9a2f3086ac6b3ebb1 2023-04-13 14:54:51,859 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/bf33403a916044e9a2f3086ac6b3ebb1, entries=150, sequenceid=301, filesize=11.9 K 2023-04-13 14:54:51,859 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for ccd2cd4c855e3a8c5393dd57d5cc737e in 1267ms, sequenceid=301, compaction requested=true 2023-04-13 14:54:51,859 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:51,860 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:51,860 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:54:51,860 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:54:51,860 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:51,860 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:51,862 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51244 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:54:51,862 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/A is initiating minor compaction (all files) 2023-04-13 14:54:51,862 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/A in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,862 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/81fbcbc04bf44564b9579f7f0fb861b0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/58b9f59613fc4e67bd90b6f1ea50190d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/598577a1df2e495c84460c16a3420717, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/270616b5d5ba40e19a8764472bce39a9] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=50.0 K 2023-04-13 14:54:51,862 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48804 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:54:51,862 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/B is initiating minor compaction (all files) 2023-04-13 14:54:51,863 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/B in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,863 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/791ffc946ee54a4c879fda2340fdfdc3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/04969316840145adbd733b32504b8841, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/835af86082e84722bbeae714bbba51fa, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/70ceef37130943959f8f83453aeeac8e] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=47.7 K 2023-04-13 14:54:51,863 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 81fbcbc04bf44564b9579f7f0fb861b0, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681397687600 2023-04-13 14:54:51,863 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 791ffc946ee54a4c879fda2340fdfdc3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681397687600 2023-04-13 14:54:51,863 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 58b9f59613fc4e67bd90b6f1ea50190d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681397688213 2023-04-13 14:54:51,863 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 04969316840145adbd733b32504b8841, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681397688213 2023-04-13 14:54:51,864 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 598577a1df2e495c84460c16a3420717, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1681397688728 2023-04-13 14:54:51,864 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 835af86082e84722bbeae714bbba51fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1681397688728 2023-04-13 14:54:51,864 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 270616b5d5ba40e19a8764472bce39a9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681397689958 2023-04-13 14:54:51,864 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 70ceef37130943959f8f83453aeeac8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681397689965 2023-04-13 14:54:51,873 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#B#compaction#230 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:51,876 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#A#compaction#231 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:51,887 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:51,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:51,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:51,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:51,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:51,907 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:51,907 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:51,907 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:51,907 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:51,907 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,908 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,911 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,911 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:51,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:51,912 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:51,912 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:51,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:51,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:51,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:51,920 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:51,920 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:51,920 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:51,925 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:51,925 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:51,925 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,925 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:51,926 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 14:54:51,926 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:51,926 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:51,926 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:51,926 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:51,926 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:51,926 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:51,939 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/39820c88dccc41198c131e2ccbe46112 2023-04-13 14:54:51,940 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/78812c8db64b4e79bad905bed164166f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/78812c8db64b4e79bad905bed164166f 2023-04-13 14:54:51,945 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/A of ccd2cd4c855e3a8c5393dd57d5cc737e into 78812c8db64b4e79bad905bed164166f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:51,945 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:51,945 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/A, priority=12, startTime=1681397691859; duration=0sec 2023-04-13 14:54:51,945 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:51,945 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:54:51,946 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48804 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:54:51,946 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/C is initiating minor compaction (all files) 2023-04-13 14:54:51,947 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/C in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:51,947 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/798053fae5a84e88ab0904b32057b12f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/f94a511f13724a54af13b996a87ca11a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/42040992c6814e23bd6b19cd534f23f9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/bf33403a916044e9a2f3086ac6b3ebb1] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=47.7 K 2023-04-13 14:54:51,947 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 798053fae5a84e88ab0904b32057b12f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681397687600 2023-04-13 14:54:51,947 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting f94a511f13724a54af13b996a87ca11a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681397688213 2023-04-13 14:54:51,948 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 42040992c6814e23bd6b19cd534f23f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1681397688728 2023-04-13 14:54:51,948 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting bf33403a916044e9a2f3086ac6b3ebb1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681397689965 2023-04-13 14:54:51,954 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/078542d46369411e81a0cbc469267178 2023-04-13 14:54:51,957 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#C#compaction#234 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:52,315 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/1c4a3cd022e54866b8a12b5bf9e2521c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/1c4a3cd022e54866b8a12b5bf9e2521c 2023-04-13 14:54:52,320 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/B of ccd2cd4c855e3a8c5393dd57d5cc737e into 1c4a3cd022e54866b8a12b5bf9e2521c(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:52,320 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:52,320 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/B, priority=12, startTime=1681397691860; duration=0sec 2023-04-13 14:54:52,320 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:52,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:52,355 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:52,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397692369, max timestamp=1681397692375Num uniques 859; 2023-04-13 14:54:52,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:52,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397692369, max timestamp=1681397692375Num uniques 859; 2023-04-13 14:54:52,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:52,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397692369, max timestamp=1681397692375Num uniques 859; 2023-04-13 14:54:52,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:52,393 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/97ca697fc4014f5f98cd2a2a58ec0f5f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/97ca697fc4014f5f98cd2a2a58ec0f5f 2023-04-13 14:54:52,394 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/0ab8f598fdf141ef9927d5615978fddb 2023-04-13 14:54:52,398 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/C of ccd2cd4c855e3a8c5393dd57d5cc737e into 97ca697fc4014f5f98cd2a2a58ec0f5f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:52,398 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:52,398 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/C, priority=12, startTime=1681397691860; duration=0sec 2023-04-13 14:54:52,398 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:52,399 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/39820c88dccc41198c131e2ccbe46112 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/39820c88dccc41198c131e2ccbe46112 2023-04-13 14:54:52,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1681397752399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,403 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/39820c88dccc41198c131e2ccbe46112, entries=150, sequenceid=310, filesize=11.9 K 2023-04-13 14:54:52,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397752399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397752400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397752401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,404 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/078542d46369411e81a0cbc469267178 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/078542d46369411e81a0cbc469267178 2023-04-13 14:54:52,408 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/078542d46369411e81a0cbc469267178, entries=150, sequenceid=310, filesize=11.9 K 2023-04-13 14:54:52,409 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/0ab8f598fdf141ef9927d5615978fddb as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/0ab8f598fdf141ef9927d5615978fddb 2023-04-13 14:54:52,412 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/0ab8f598fdf141ef9927d5615978fddb, entries=150, sequenceid=310, filesize=11.9 K 2023-04-13 14:54:52,413 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for ccd2cd4c855e3a8c5393dd57d5cc737e in 488ms, sequenceid=310, compaction requested=false 2023-04-13 14:54:52,413 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:52,413 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:52,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:52,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:52,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:52,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,417 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:52,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:52,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:52,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:52,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:52,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:52,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@274f9339[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:52,419 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:52,420 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,421 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,421 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:52,421 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:52,421 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:52,421 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:52,422 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,422 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:52,423 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:52,423 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:52,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:52,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:52,425 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:52,425 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:52,425 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:52,425 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:52,425 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:52,430 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,430 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:52,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,430 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:52,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:52,431 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,432 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,434 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,434 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,434 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,434 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,434 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:52,434 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:52,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:52,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:52,434 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:52,434 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:52,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:52,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:52,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:52,434 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,435 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:52,435 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,435 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:52,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:52,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7683, totalHeapSize=17712, min timestamp=1681397692389, max timestamp=1681397692505Num uniques 1068; 2023-04-13 14:54:52,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48903, new segment data size=7683, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-13 14:54:52,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397692389, max timestamp=1681397692505Num uniques 1068; 2023-04-13 14:54:52,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7682, totalHeapSize=17712, min timestamp=1681397692389, max timestamp=1681397692505Num uniques 1068; 2023-04-13 14:54:52,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-13 14:54:52,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48902, new segment data size=7682, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-13 14:54:52,508 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=33.54 KB heapSize=85.31 KB 2023-04-13 14:54:52,509 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:52,509 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:52,509 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:52,509 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:52,509 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:52,509 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:52,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397692520, max timestamp=1681397692528Num uniques 859; 2023-04-13 14:54:52,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:52,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397692520, max timestamp=1681397692528Num uniques 859; 2023-04-13 14:54:52,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:52,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397692520, max timestamp=1681397692528Num uniques 859; 2023-04-13 14:54:52,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:52,535 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:52,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:52,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:52,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:52,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:52,539 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:52,539 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:52,539 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:52,539 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:52,539 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,539 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,540 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,540 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:52,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:52,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:52,541 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,541 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:52,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:52,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:52,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:52,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:52,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:52,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:52,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:52,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:52,542 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:52,542 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:52,542 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:52,542 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:52,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:52,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:52,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:52,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:52,543 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,544 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:52,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:52,544 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,544 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:52,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:52,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:52,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:52,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66b924df[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:52,546 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:52,546 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,547 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,548 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:52,548 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:52,548 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:52,548 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,548 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:52,549 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,549 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:52,549 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,549 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:52,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:52,552 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:52,552 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:52,552 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:52,552 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:52,552 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:52,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681397692536, max timestamp=1681397692552Num uniques 1068; 2023-04-13 14:54:52,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681397692536, max timestamp=1681397692552Num uniques 1068; 2023-04-13 14:54:52,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-13 14:54:52,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-13 14:54:52,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681397692536, max timestamp=1681397692552Num uniques 1068; 2023-04-13 14:54:52,555 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-13 14:54:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:52,555 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,555 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:52,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:52,555 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,557 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,562 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:52,562 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:52,562 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:52,562 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:52,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:52,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:52,562 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,562 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,562 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,562 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,562 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,562 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:52,562 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,562 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:52,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:52,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397752591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397752592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397752592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397752592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,662 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:52,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:52,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:52,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:52,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:52,666 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:52,666 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:52,666 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:52,666 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:52,666 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,666 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,667 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:52,667 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:52,668 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,668 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:52,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:52,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:52,669 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:52,669 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:52,669 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:52,669 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:52,669 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:52,669 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:52,669 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:52,669 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:52,669 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:52,669 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,671 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:52,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:52,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:52,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:52,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51a69016[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:52,673 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:52,673 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,674 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:52,674 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,674 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:52,676 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:52,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:52,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:52,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:52,679 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:52,679 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:52,679 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:52,679 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:52,679 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:52,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:52,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:52,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:52,681 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,682 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,684 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:52,684 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:52,685 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:52,684 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:52,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:52,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:52,684 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,685 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,685 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,685 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,685 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:52,685 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:52,685 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,685 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:52,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397752695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397752697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397752697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397752697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53214 deadline: 1681397752749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,785 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:52,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:52,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:52,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:52,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:52,789 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:52,789 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:52,789 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:52,789 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:52,789 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,789 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,790 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,790 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:52,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:52,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:52,791 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,791 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:52,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:52,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:52,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:52,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:52,792 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:52,792 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:52,792 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:52,793 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:52,793 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:52,793 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:52,793 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:52,793 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:52,793 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:52,793 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:52,793 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:52,793 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,793 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:52,793 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:52,795 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,796 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:52,796 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,796 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:52,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:52,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5653cda6[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:52,797 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:52,797 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,798 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,799 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,799 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:52,799 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:52,800 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,800 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:52,800 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:52,800 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:52,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:52,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:52,802 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:52,802 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:52,802 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:52,802 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:52,803 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:52,804 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,804 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:52,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,804 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:52,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:52,804 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,806 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,809 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:52,809 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:52,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:52,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:52,809 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:52,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:52,809 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:52,809 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:52,809 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,809 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:52,809 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:52,810 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:52,810 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,810 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,810 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397752897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397752901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397752902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:52,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397752902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,910 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:52,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:52,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:52,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:52,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:52,913 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:52,913 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:52,913 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:52,913 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:52,913 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,913 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,915 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,915 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:52,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:52,915 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,915 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:52,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:52,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:52,917 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:52,917 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:52,917 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:52,917 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:52,917 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:52,917 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:52,917 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,917 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:52,917 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:52,918 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,918 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:52,919 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,919 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:52,919 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/60b57fdada294bd3a1c6e65707637af4 2023-04-13 14:54:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:52,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:52,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:52,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dde5a0e[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:52,923 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:52,923 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,924 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,924 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:52,924 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:52,924 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:52,924 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,924 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:52,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:52,930 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:52,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:52,933 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:52,933 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:52,933 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:52,933 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:52,933 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:52,935 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,935 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,935 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:52,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,935 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:52,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,936 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,941 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:52,941 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,941 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:52,941 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:52,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:52,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:52,941 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:52,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:52,941 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:52,941 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:52,942 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:52,941 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:52,942 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,942 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:52,942 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:52,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:52,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:53,042 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:53,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:53,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:53,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:53,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:53,045 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:53,045 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:53,045 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:53,045 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:53,045 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,045 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,048 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:53,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:53,048 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,048 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,049 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:53,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:53,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:53,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:53,051 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:53,051 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:53,051 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:53,051 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:53,053 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,053 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,053 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,053 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:53,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@226e08b6[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:53,055 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:53,055 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,056 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,056 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:53,056 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:53,056 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:53,056 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,056 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:53,057 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:53,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:53,057 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:53,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:53,060 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:53,060 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:53,060 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:53,060 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:53,060 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:53,066 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,066 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,066 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:53,066 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,068 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,070 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,070 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,070 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,070 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:53,070 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:53,070 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:53,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:53,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:53,070 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,070 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,070 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,070 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:53,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:53,070 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:53,070 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:53,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:53,171 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:53,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:53,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:53,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:53,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:53,174 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:53,174 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:53,174 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:53,174 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:53,174 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,174 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,175 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:53,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:53,176 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,176 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,176 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:53,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:53,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:53,177 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:53,177 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:53,177 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:53,177 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:53,179 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,179 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,179 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,179 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:53,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@472d1acd[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:53,181 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:53,181 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,183 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:53,184 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:53,185 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:53,185 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:53,185 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:53,185 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:53,185 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:53,185 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:53,185 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:53,185 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:53,187 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,187 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,187 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:53,188 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,189 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,191 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,191 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,191 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:53,191 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:53,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:53,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:53,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:53,192 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,192 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,192 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,192 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,192 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:53,192 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:53,192 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:53,192 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:53,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:53,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:53,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397753199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:53,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397753202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:53,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397753204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:53,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397753207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,292 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:53,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:53,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:53,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:53,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:53,296 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:53,296 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:53,296 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:53,296 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:53,296 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,296 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,298 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,298 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:53,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:53,299 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,299 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:53,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:53,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:53,300 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:53,300 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:53,300 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:53,300 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:53,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:53,302 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,302 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,302 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,303 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:53,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f5bc173[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:53,304 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:53,304 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,306 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,306 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:53,306 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:53,306 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:53,306 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,306 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:53,307 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:53,307 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,307 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:53,308 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:53,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:53,311 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:53,311 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:53,311 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:53,311 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:53,311 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:53,316 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,316 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,316 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:53,317 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:53,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,318 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,320 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:53,320 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:53,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:53,320 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:53,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:53,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:53,320 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:53,320 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,320 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,320 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:53,321 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:53,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:53,321 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:53,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:53,320 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,321 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,321 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,332 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/5d1055f94c4a41d0beb0a0c9e76e2fe5 2023-04-13 14:54:53,421 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:53,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:53,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:53,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:53,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:53,425 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:53,425 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:53,425 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:53,425 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:53,425 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,425 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,429 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:53,429 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,430 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,430 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:53,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:53,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:53,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:53,431 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:53,431 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:53,431 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:53,432 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:53,432 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:53,432 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:53,432 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:53,437 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,437 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,438 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,438 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:53,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@702af4bc[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:53,440 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:53,440 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,441 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,441 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:53,441 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:53,441 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:53,441 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,441 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:53,442 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,442 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:53,442 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,442 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:53,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:53,445 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:53,445 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:53,445 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:53,445 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:53,445 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:53,446 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,446 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,446 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:53,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,447 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,448 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,450 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,450 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,450 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,450 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:53,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:53,450 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:53,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:53,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:53,450 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,451 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,451 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,451 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:53,451 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:53,451 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:53,451 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:53,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:53,551 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:53,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:53,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:53,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:53,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:53,554 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:53,554 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:53,554 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:53,554 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:53,555 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,555 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,556 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:53,556 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:53,557 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,557 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:53,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:53,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:53,558 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:53,558 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:53,558 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:53,559 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:53,559 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:53,559 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:53,559 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,559 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:53,559 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:53,560 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,561 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,561 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:53,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22d379b7[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:53,562 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:53,562 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,563 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,563 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:53,563 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:53,563 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:53,564 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,564 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:53,566 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,566 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:53,566 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,566 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:53,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:53,568 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:53,568 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:53,568 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:53,568 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:53,568 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:53,570 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,570 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,570 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:53,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:53,571 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,572 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,574 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,574 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:53,574 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:53,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:53,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:53,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:53,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:53,574 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:53,574 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,575 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,575 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,575 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,575 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,575 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:53,575 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:53,575 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:53,675 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:53,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:53,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:53,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:53,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:53,678 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:53,678 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:53,678 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:53,678 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:53,679 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,679 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,680 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:53,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:53,680 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,680 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:53,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:53,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:53,681 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:53,681 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:53,681 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:53,682 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:53,682 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:53,682 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:53,682 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:53,687 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,687 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,687 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,688 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d87f452[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:53,689 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:53,689 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,690 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,690 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:53,690 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,690 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:53,690 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:53,690 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:53,692 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:53,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:53,694 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:53,694 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:53,694 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:53,694 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:53,694 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:53,696 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,696 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,696 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:53,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:53,696 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,697 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:53,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53236 deadline: 1681397753703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,704 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,704 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:53,704 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:53,704 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:53,704 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,705 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,705 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,704 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:53,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:53,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:53,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:53,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:53,705 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:53,705 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,705 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:53,705 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:53,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:53,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397753706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:53,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53292 deadline: 1681397753709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:53,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397753710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,741 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/f48b99e3ca6b49999ef7c6cf8eaada67 2023-04-13 14:54:53,746 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/60b57fdada294bd3a1c6e65707637af4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/60b57fdada294bd3a1c6e65707637af4 2023-04-13 14:54:53,749 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/60b57fdada294bd3a1c6e65707637af4, entries=150, sequenceid=354, filesize=11.9 K 2023-04-13 14:54:53,750 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/5d1055f94c4a41d0beb0a0c9e76e2fe5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5d1055f94c4a41d0beb0a0c9e76e2fe5 2023-04-13 14:54:53,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5d1055f94c4a41d0beb0a0c9e76e2fe5, entries=150, sequenceid=354, filesize=11.9 K 2023-04-13 14:54:53,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/f48b99e3ca6b49999ef7c6cf8eaada67 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/f48b99e3ca6b49999ef7c6cf8eaada67 2023-04-13 14:54:53,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/f48b99e3ca6b49999ef7c6cf8eaada67, entries=150, sequenceid=354, filesize=11.9 K 2023-04-13 14:54:53,758 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~125.11 KB/128112, currentSize=147.60 KB/151140 for ccd2cd4c855e3a8c5393dd57d5cc737e in 1250ms, sequenceid=354, compaction requested=true 2023-04-13 14:54:53,758 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:53,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:53,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:53,758 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:53,758 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:53,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:53,759 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:53,759 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/A is initiating minor compaction (all files) 2023-04-13 14:54:53,759 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:53,759 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/A in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:53,759 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/B is initiating minor compaction (all files) 2023-04-13 14:54:53,759 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/78812c8db64b4e79bad905bed164166f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/39820c88dccc41198c131e2ccbe46112, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/60b57fdada294bd3a1c6e65707637af4] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=36.2 K 2023-04-13 14:54:53,759 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/B in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:53,759 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/1c4a3cd022e54866b8a12b5bf9e2521c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/078542d46369411e81a0cbc469267178, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5d1055f94c4a41d0beb0a0c9e76e2fe5] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=36.2 K 2023-04-13 14:54:53,760 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 78812c8db64b4e79bad905bed164166f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681397689965 2023-04-13 14:54:53,760 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 1c4a3cd022e54866b8a12b5bf9e2521c, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681397689965 2023-04-13 14:54:53,760 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 39820c88dccc41198c131e2ccbe46112, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1681397690601 2023-04-13 14:54:53,760 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 078542d46369411e81a0cbc469267178, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1681397690601 2023-04-13 14:54:53,760 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 60b57fdada294bd3a1c6e65707637af4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1681397692394 2023-04-13 14:54:53,760 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 5d1055f94c4a41d0beb0a0c9e76e2fe5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1681397692394 2023-04-13 14:54:53,766 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#B#compaction#240 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:53,767 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#A#compaction#239 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:53,805 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:53,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:53,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:53,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:53,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:53,808 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:53,808 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:53,808 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:53,808 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:53,809 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,809 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,810 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:53,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:53,810 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,810 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:53,810 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:53,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:53,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:53,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:53,811 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:53,811 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:53,811 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:53,812 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:53,812 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:53,812 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:53,812 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:53,812 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=147.60 KB heapSize=409.92 KB 2023-04-13 14:54:53,812 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:53,812 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:53,812 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:53,812 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:53,812 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:53,812 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:53,821 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/7434e990e8134659812197b95da8c37e 2023-04-13 14:54:54,175 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/7642f44c4e1342d4a9e0816f64100dd0 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/7642f44c4e1342d4a9e0816f64100dd0 2023-04-13 14:54:54,180 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/B of ccd2cd4c855e3a8c5393dd57d5cc737e into 7642f44c4e1342d4a9e0816f64100dd0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:54,180 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:54,180 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/B, priority=13, startTime=1681397693758; duration=0sec 2023-04-13 14:54:54,180 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:54,180 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:54,181 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:54,182 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/C is initiating minor compaction (all files) 2023-04-13 14:54:54,182 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/C in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:54,182 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/97ca697fc4014f5f98cd2a2a58ec0f5f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/0ab8f598fdf141ef9927d5615978fddb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/f48b99e3ca6b49999ef7c6cf8eaada67] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=36.2 K 2023-04-13 14:54:54,182 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 97ca697fc4014f5f98cd2a2a58ec0f5f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681397689965 2023-04-13 14:54:54,183 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 0ab8f598fdf141ef9927d5615978fddb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1681397690601 2023-04-13 14:54:54,183 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting f48b99e3ca6b49999ef7c6cf8eaada67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1681397692394 2023-04-13 14:54:54,183 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/fe2a02ce9f1e4d5980663411ad0e419f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/fe2a02ce9f1e4d5980663411ad0e419f 2023-04-13 14:54:54,187 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/A of ccd2cd4c855e3a8c5393dd57d5cc737e into fe2a02ce9f1e4d5980663411ad0e419f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:54,187 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:54,187 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/A, priority=13, startTime=1681397693758; duration=0sec 2023-04-13 14:54:54,188 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:54,195 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#C#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:54,233 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/2674bb3343b2454d9178bcc7e1519eb3 2023-04-13 14:54:54,604 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/7c4872f6cde04cf684f019d1bdc49e2e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7c4872f6cde04cf684f019d1bdc49e2e 2023-04-13 14:54:54,609 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/C of ccd2cd4c855e3a8c5393dd57d5cc737e into 7c4872f6cde04cf684f019d1bdc49e2e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:54,609 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:54,609 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/C, priority=13, startTime=1681397693758; duration=0sec 2023-04-13 14:54:54,609 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:54,644 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/7bf83d13637945b5aa4da2eb49076a75 2023-04-13 14:54:54,648 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/7434e990e8134659812197b95da8c37e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/7434e990e8134659812197b95da8c37e 2023-04-13 14:54:54,651 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/7434e990e8134659812197b95da8c37e, entries=150, sequenceid=407, filesize=11.9 K 2023-04-13 14:54:54,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/2674bb3343b2454d9178bcc7e1519eb3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/2674bb3343b2454d9178bcc7e1519eb3 2023-04-13 14:54:54,656 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/2674bb3343b2454d9178bcc7e1519eb3, entries=150, sequenceid=407, filesize=11.9 K 2023-04-13 14:54:54,657 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/7bf83d13637945b5aa4da2eb49076a75 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7bf83d13637945b5aa4da2eb49076a75 2023-04-13 14:54:54,662 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7bf83d13637945b5aa4da2eb49076a75, entries=150, sequenceid=407, filesize=11.9 K 2023-04-13 14:54:54,662 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~409.88 KB/419712, currentSize=0 B/0 for ccd2cd4c855e3a8c5393dd57d5cc737e in 850ms, sequenceid=407, compaction requested=false 2023-04-13 14:54:54,663 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:54,663 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:54,664 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,665 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,665 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:54,665 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c86ecc8[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:54,667 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:54,667 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,669 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,669 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:54,669 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:54,669 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:54,669 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,669 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:54,671 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,671 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:54,671 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:54,671 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:54,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:54,674 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:54,674 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:54,674 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:54,674 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:54,674 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:54,676 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,676 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,676 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:54,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,677 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,678 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:54,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:54,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:54,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:54,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,680 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:54,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:54,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:54,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:54,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,681 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:54,681 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:54,681 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:54,681 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:54,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:54,716 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:54:54,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:54,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:54,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:54,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:54,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:54,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:54,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397694726, max timestamp=1681397694735Num uniques 859; 2023-04-13 14:54:54,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:54:54,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397694726, max timestamp=1681397694735Num uniques 859; 2023-04-13 14:54:54,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397694726, max timestamp=1681397694735Num uniques 859; 2023-04-13 14:54:54,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:54:54,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:54:54,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:54,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397754747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:54,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53274 deadline: 1681397754750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:54,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397754751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:54,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397754752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:54,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53214 deadline: 1681397754771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,774 DEBUG [Thread-1474] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:54:54,781 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:54,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:54,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:54,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:54,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:54,784 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:54,784 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:54,784 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:54,784 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:54,785 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,785 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:54,786 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,786 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:54,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:54,786 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,786 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:54,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:54,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:54,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:54,787 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:54,787 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:54,787 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:54,788 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:54,788 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:54,788 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:54,788 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:54,788 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:54,788 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:54,788 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:54,788 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:54,788 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,788 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:54,788 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:54,793 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,793 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:54,794 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,794 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bae6cf0[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:54,796 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:54,796 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,797 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:54,797 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,797 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:54,798 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:54,798 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:54,798 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,798 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:54,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:54,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:54,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:54,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:54,801 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:54,801 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:54,801 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:54,801 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:54,801 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:54,803 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,803 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:54,803 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:54,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:54,803 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,804 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,808 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:54,808 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:54,808 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:54,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:54,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:54,808 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:54,808 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:54,808 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,808 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,808 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,808 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,808 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:54,808 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:54,808 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:54,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:54,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:54,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397754853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:54,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397754853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:54,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397754856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:54,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397754856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,908 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:54,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:54,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:54,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:54,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:54,912 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:54,912 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:54,912 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:54,912 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:54,912 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,912 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:54,913 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,913 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:54,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:54,914 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,914 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:54,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:54,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:54,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:54,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:54,915 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:54,915 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:54,915 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:54,915 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:54,915 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:54,915 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:54,915 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:54,915 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:54,915 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:54,915 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:54,915 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:54,916 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,916 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:54,916 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:54,917 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,917 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:54,917 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,918 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:54,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:54,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4209bcf3[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:54,919 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:54,919 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,920 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,920 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:54,920 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:54,920 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,920 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:54,920 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:54,923 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,923 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:54,923 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:54,923 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:54,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:54,925 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:54,925 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:54,925 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:54,925 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:54,925 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:54,927 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,927 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,927 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:54,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:54,927 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,929 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,931 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,931 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:54,931 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:54,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:54,931 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:54,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:54,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:54,931 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:54,931 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:54,932 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:54,932 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,932 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:54,932 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:54,932 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:54,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:54,932 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:54,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:55,032 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:55,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:55,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:55,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:55,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:55,035 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:55,035 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:55,035 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:55,035 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:55,036 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,036 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,037 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:55,037 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:55,037 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,037 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:55,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:55,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:55,038 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:55,038 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:55,038 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:55,039 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:55,039 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:55,039 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:55,039 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:55,039 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:55,039 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:55,039 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:55,039 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:55,039 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,039 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:55,039 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:55,042 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@709ee584[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:55,043 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:55,043 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,044 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,045 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:55,045 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:55,045 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:55,045 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,045 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:55,049 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,049 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:55,049 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:55,049 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:55,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:55,051 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:55,051 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:55,051 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:55,051 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:55,052 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:55,054 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,054 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:55,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,054 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:55,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:55,055 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,056 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397755056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397755056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,058 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,059 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,059 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,059 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:55,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:55,059 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:55,059 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,059 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:55,059 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:55,059 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:55,059 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:55,059 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:55,059 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:55,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:55,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397755060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397755060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/8261670a87524e478e7782fa44ce1bd0 2023-04-13 14:54:55,159 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:55,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:55,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:55,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:55,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:55,162 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:55,162 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:55,162 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:55,162 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:55,162 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,162 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,164 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,164 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:55,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:55,164 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,164 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:55,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:55,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:55,165 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:55,165 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:55,165 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:55,166 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:55,166 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:55,166 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:55,166 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:55,166 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:55,166 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:55,166 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:55,166 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:55,166 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,166 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:55,166 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:55,167 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:55,167 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:55,168 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,168 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:55,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:55,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b95a7f3[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:55,169 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:55,169 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,171 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,171 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:55,171 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,171 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:55,171 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:55,171 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:55,172 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:55,172 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:55,172 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:55,173 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:55,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:55,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:55,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:55,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:55,175 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:55,175 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:55,175 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:55,175 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:55,175 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:55,176 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,176 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:55,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,176 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:55,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:55,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:55,177 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:55,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,178 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:55,180 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:55,180 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,180 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:55,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:55,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:55,180 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:55,180 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:55,181 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:55,181 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:55,181 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,181 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,181 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,181 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:55,181 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:55,281 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:55,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:55,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:55,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:55,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:55,284 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:55,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:55,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:55,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:55,285 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,285 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,287 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,287 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:55,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:55,287 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,288 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:55,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:55,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:55,289 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:55,289 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:55,289 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:55,289 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:55,291 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:55,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:55,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1354919d[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:55,293 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:55,293 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,294 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:55,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:55,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:55,294 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,295 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:55,296 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:55,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:55,298 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:55,298 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:55,298 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:55,298 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:55,298 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:55,301 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,301 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:55,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,301 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:55,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:55,302 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,303 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,304 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:55,304 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:55,305 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:55,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:55,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:55,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:55,305 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,305 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,305 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,305 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,305 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,305 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,305 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:55,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:55,305 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:55,305 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:55,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397755360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397755361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397755362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397755363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,405 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:55,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:55,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:55,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:55,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:55,409 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:55,409 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:55,409 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:55,409 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:55,409 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,409 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,411 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:55,411 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,411 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,411 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:55,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:55,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:55,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:55,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:55,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:55,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:55,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:55,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:55,413 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:55,413 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:55,413 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:55,413 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:55,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:55,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:55,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:55,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:55,415 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:55,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:55,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:55,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:55,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c354920[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:55,416 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:55,416 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,419 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:55,419 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,419 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:55,420 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:55,423 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:55,423 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:55,423 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:55,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:55,423 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:55,423 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:55,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:55,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:55,425 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,426 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,431 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:55,432 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:55,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:55,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:55,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:55,432 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:55,432 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:55,432 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,432 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,432 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,432 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:55,432 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,432 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:55,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:55,432 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,432 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,532 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:55,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:55,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:55,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:55,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:55,536 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:55,536 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:55,536 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:55,536 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:55,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,538 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,538 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:55,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:55,538 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,538 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:55,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:55,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:55,540 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:55,540 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:55,540 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:55,540 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:55,542 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:55,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b003de7[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:55,543 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:55,544 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,545 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:55,545 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,545 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:55,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/f35b375a12f0490fae713e34394e7bf3 2023-04-13 14:54:55,547 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:55,547 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:55,547 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:55,547 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:55,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:55,551 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:55,551 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:55,551 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:55,551 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:55,551 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:55,552 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,552 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:55,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,552 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:55,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,554 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,555 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,557 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/24f1ac290c0b4a02a15e03ec0e0a2b3e 2023-04-13 14:54:55,557 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,557 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,557 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,557 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,557 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:55,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:55,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:55,557 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,558 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:55,558 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,557 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:55,558 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:55,558 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:55,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:55,558 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:55,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:55,563 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/8261670a87524e478e7782fa44ce1bd0 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/8261670a87524e478e7782fa44ce1bd0 2023-04-13 14:54:55,567 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/8261670a87524e478e7782fa44ce1bd0, entries=150, sequenceid=421, filesize=11.9 K 2023-04-13 14:54:55,567 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/f35b375a12f0490fae713e34394e7bf3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f35b375a12f0490fae713e34394e7bf3 2023-04-13 14:54:55,571 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f35b375a12f0490fae713e34394e7bf3, entries=150, sequenceid=421, filesize=11.9 K 2023-04-13 14:54:55,572 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/24f1ac290c0b4a02a15e03ec0e0a2b3e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/24f1ac290c0b4a02a15e03ec0e0a2b3e 2023-04-13 14:54:55,575 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/24f1ac290c0b4a02a15e03ec0e0a2b3e, entries=150, sequenceid=421, filesize=11.9 K 2023-04-13 14:54:55,576 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ccd2cd4c855e3a8c5393dd57d5cc737e in 860ms, sequenceid=421, compaction requested=true 2023-04-13 14:54:55,576 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:55,576 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:55,576 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:55,576 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:55,576 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:55,576 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:55,577 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:55,577 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/A is initiating minor compaction (all files) 2023-04-13 14:54:55,577 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:55,577 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/A in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:55,577 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/B is initiating minor compaction (all files) 2023-04-13 14:54:55,577 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/B in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:55,577 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/fe2a02ce9f1e4d5980663411ad0e419f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/7434e990e8134659812197b95da8c37e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/8261670a87524e478e7782fa44ce1bd0] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=36.3 K 2023-04-13 14:54:55,578 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/7642f44c4e1342d4a9e0816f64100dd0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/2674bb3343b2454d9178bcc7e1519eb3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f35b375a12f0490fae713e34394e7bf3] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=36.3 K 2023-04-13 14:54:55,578 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting fe2a02ce9f1e4d5980663411ad0e419f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1681397692394 2023-04-13 14:54:55,578 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 7642f44c4e1342d4a9e0816f64100dd0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1681397692394 2023-04-13 14:54:55,579 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 7434e990e8134659812197b95da8c37e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1681397692586 2023-04-13 14:54:55,579 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 2674bb3343b2454d9178bcc7e1519eb3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1681397692586 2023-04-13 14:54:55,579 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 8261670a87524e478e7782fa44ce1bd0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1681397694711 2023-04-13 14:54:55,579 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting f35b375a12f0490fae713e34394e7bf3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1681397694711 2023-04-13 14:54:55,587 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#B#compaction#248 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:55,587 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#A#compaction#249 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:55,596 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/bb99fc423cb74a0788381bb5e1620d90 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/bb99fc423cb74a0788381bb5e1620d90 2023-04-13 14:54:55,600 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/A of ccd2cd4c855e3a8c5393dd57d5cc737e into bb99fc423cb74a0788381bb5e1620d90(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:55,600 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:55,600 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/A, priority=13, startTime=1681397695576; duration=0sec 2023-04-13 14:54:55,600 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:55,600 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:55,601 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:55,601 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/C is initiating minor compaction (all files) 2023-04-13 14:54:55,602 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/C in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:55,602 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7c4872f6cde04cf684f019d1bdc49e2e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7bf83d13637945b5aa4da2eb49076a75, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/24f1ac290c0b4a02a15e03ec0e0a2b3e] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=36.3 K 2023-04-13 14:54:55,602 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 7c4872f6cde04cf684f019d1bdc49e2e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1681397692394 2023-04-13 14:54:55,602 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 7bf83d13637945b5aa4da2eb49076a75, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1681397692586 2023-04-13 14:54:55,603 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 24f1ac290c0b4a02a15e03ec0e0a2b3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1681397694711 2023-04-13 14:54:55,609 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#C#compaction#250 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:55,658 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:55,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:55,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:55,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:55,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:55,661 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:55,661 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:55,661 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:55,661 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:55,662 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,662 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,663 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,663 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:55,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:55,664 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,664 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:55,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:55,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:55,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:55,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:55,665 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:55,665 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:55,665 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:55,665 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:55,665 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:55,665 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:55,665 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:55,666 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-13 14:54:55,666 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:55,666 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:55,666 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:55,666 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:55,666 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:55,666 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:55,676 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/9761eb771b814f0a8a936eeebdce9ee6 2023-04-13 14:54:55,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:55,865 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:55,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397755869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397755870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397755871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397755873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397755973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397755974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397755974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:55,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397755976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:55,996 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/d99bfcffcac9424bb2c80fd7d0ce3457 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/d99bfcffcac9424bb2c80fd7d0ce3457 2023-04-13 14:54:56,000 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/B of ccd2cd4c855e3a8c5393dd57d5cc737e into d99bfcffcac9424bb2c80fd7d0ce3457(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:56,000 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:56,000 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/B, priority=13, startTime=1681397695576; duration=0sec 2023-04-13 14:54:56,000 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:56,017 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/b1371e749f914c858db51b2bce0abae3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/b1371e749f914c858db51b2bce0abae3 2023-04-13 14:54:56,021 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/C of ccd2cd4c855e3a8c5393dd57d5cc737e into b1371e749f914c858db51b2bce0abae3(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:56,021 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:56,021 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/C, priority=13, startTime=1681397695576; duration=0sec 2023-04-13 14:54:56,021 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:56,093 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/0c88a3acb15046ffba3c1f2225c5af4c 2023-04-13 14:54:56,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:56,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397756177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:56,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397756177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:56,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397756178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:56,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397756178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:56,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397756478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:56,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397756481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:56,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397756482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:56,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397756482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,502 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/4e066b3ddd4344b7a282ec4493c28079 2023-04-13 14:54:56,507 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/9761eb771b814f0a8a936eeebdce9ee6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/9761eb771b814f0a8a936eeebdce9ee6 2023-04-13 14:54:56,510 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/9761eb771b814f0a8a936eeebdce9ee6, entries=150, sequenceid=456, filesize=11.9 K 2023-04-13 14:54:56,513 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/0c88a3acb15046ffba3c1f2225c5af4c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0c88a3acb15046ffba3c1f2225c5af4c 2023-04-13 14:54:56,516 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0c88a3acb15046ffba3c1f2225c5af4c, entries=150, sequenceid=456, filesize=11.9 K 2023-04-13 14:54:56,517 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/4e066b3ddd4344b7a282ec4493c28079 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/4e066b3ddd4344b7a282ec4493c28079 2023-04-13 14:54:56,521 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/4e066b3ddd4344b7a282ec4493c28079, entries=150, sequenceid=456, filesize=11.9 K 2023-04-13 14:54:56,521 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for ccd2cd4c855e3a8c5393dd57d5cc737e in 856ms, sequenceid=456, compaction requested=false 2023-04-13 14:54:56,521 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:56,521 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:56,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,527 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:56,527 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:56,527 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:56,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:56,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d9cbb4b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:56,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:56,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:56,530 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:56,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:56,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:56,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:56,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:56,530 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,530 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:56,531 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,531 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:56,532 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:56,532 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:56,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:56,534 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:56,534 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:56,534 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:56,534 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:56,534 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:56,536 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:56,536 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:56,536 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:56,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:56,536 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:56,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:56,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:56,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:56,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:56,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:56,541 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:56,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:56,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:56,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:56,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:56,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:56,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:56,541 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:56,541 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:56,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:56,641 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:56,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:56,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:56,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:56,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:56,645 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:56,645 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:56,645 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:56,645 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:56,645 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:56,645 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:56,647 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,647 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:56,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:56,647 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:56,647 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:56,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:56,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:56,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:56,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:56,648 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:56,648 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:56,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:56,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:56,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:56,649 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:56,649 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:56,649 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 14:54:56,649 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:56,649 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:56,649 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:56,649 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:56,649 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:56,649 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:56,672 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/5de07c243e3d44df92458febe5ceaf52 2023-04-13 14:54:56,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:56,983 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:57,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397696989, max timestamp=1681397697001Num uniques 859; 2023-04-13 14:54:57,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:57,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397696989, max timestamp=1681397697001Num uniques 859; 2023-04-13 14:54:57,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:57,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397696989, max timestamp=1681397697001Num uniques 859; 2023-04-13 14:54:57,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:57,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397757014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397757015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397757017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397757019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,083 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/58aecf132b8848ad952b28a74f9b87e6 2023-04-13 14:54:57,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397757119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397757120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397757120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397757122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397757323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397757323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397757325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397757325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,493 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/dec2dd11cc3646f78d2fbf60c2ad3762 2023-04-13 14:54:57,498 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/5de07c243e3d44df92458febe5ceaf52 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/5de07c243e3d44df92458febe5ceaf52 2023-04-13 14:54:57,502 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/5de07c243e3d44df92458febe5ceaf52, entries=150, sequenceid=469, filesize=11.9 K 2023-04-13 14:54:57,503 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/58aecf132b8848ad952b28a74f9b87e6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/58aecf132b8848ad952b28a74f9b87e6 2023-04-13 14:54:57,507 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/58aecf132b8848ad952b28a74f9b87e6, entries=150, sequenceid=469, filesize=11.9 K 2023-04-13 14:54:57,516 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/dec2dd11cc3646f78d2fbf60c2ad3762 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/dec2dd11cc3646f78d2fbf60c2ad3762 2023-04-13 14:54:57,520 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/dec2dd11cc3646f78d2fbf60c2ad3762, entries=150, sequenceid=469, filesize=11.9 K 2023-04-13 14:54:57,520 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ccd2cd4c855e3a8c5393dd57d5cc737e in 871ms, sequenceid=469, compaction requested=true 2023-04-13 14:54:57,521 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:57,521 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:57,521 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:57,521 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:57,521 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,521 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:57,521 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:57,522 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:57,523 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,523 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:57,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:57,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a730f1c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:57,526 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:57,526 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,527 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,527 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:57,527 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:57,527 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:57,527 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:57,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:57,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:57,530 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:57,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:57,532 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:57,532 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:57,532 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:57,533 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:57,533 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:57,534 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,534 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,534 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:57,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,535 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:57,536 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:57,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:57,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:57,541 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:57,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:57,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:57,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:57,541 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:57,542 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:57,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:57,630 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-13 14:54:57,631 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:57,631 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:57,631 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:57,631 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:57,631 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:57,631 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:54:57,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397757635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397757635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397757636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397757637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,642 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:57,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:57,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:57,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:57,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:57,645 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:57,645 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:57,645 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:57,645 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:57,645 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,645 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,647 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:57,648 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:57,648 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,648 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:57,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:57,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:57,649 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:57,649 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:57,649 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:57,649 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:57,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,650 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:57,650 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:57,651 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:57,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:57,653 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:57,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cb9293b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:57,653 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,654 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:57,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:57,654 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,654 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:57,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:57,655 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,655 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:57,655 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:57,655 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:57,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:57,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:57,657 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:57,657 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:57,657 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:57,658 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:57,658 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:57,660 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,660 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:57,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,660 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:57,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:57,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:57,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,662 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:57,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,663 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,665 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,665 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,665 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,665 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:57,665 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:57,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:57,665 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,665 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,665 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:57,665 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:57,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:57,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:57,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:57,665 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,666 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:57,666 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:57,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397757738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397757739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397757740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397757741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,766 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:57,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:57,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:57,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:57,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:57,769 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:57,769 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:57,769 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:57,769 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:57,769 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,769 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,770 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:57,770 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:57,771 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,771 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:57,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:57,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:57,772 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:57,772 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:57,772 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:57,772 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:57,773 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,773 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:57,774 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,774 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:57,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:57,775 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:57,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ccc64e[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:57,775 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,778 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,779 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:57,779 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:57,779 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,779 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:57,779 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:57,780 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,780 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:57,780 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:57,780 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:57,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:57,782 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:57,782 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:57,782 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:57,782 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:57,782 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:57,784 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,784 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:57,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,784 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:57,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:57,784 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,785 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:57,787 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:57,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:57,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:57,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:57,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:57,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:57,788 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:57,788 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,788 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:57,788 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,788 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:57,788 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:57,888 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:57,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:57,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:57,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:57,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:57,891 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:57,891 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:57,892 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:57,892 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:57,892 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,892 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,894 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:57,895 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:57,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:57,895 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,895 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:57,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:57,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:57,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:57,896 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:57,896 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:57,896 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:57,896 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:57,896 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:57,896 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:57,896 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:57,896 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:57,896 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:57,896 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:57,896 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:57,896 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,896 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:57,897 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:57,898 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,898 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:57,899 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,899 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:57,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:57,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ec2753f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:57,900 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:57,900 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,901 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,901 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:57,901 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:57,901 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:57,901 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,901 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:57,902 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,903 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:57,903 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:57,903 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:57,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:57,905 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:57,905 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:57,905 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:57,905 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:57,905 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:57,907 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,907 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,908 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:57,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:57,908 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:57,909 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:57,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:57,911 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,911 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:57,911 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:57,911 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,911 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:57,911 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,911 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,911 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:57,911 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:57,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:57,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:57,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:57,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:57,912 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:57,912 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:57,912 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:57,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:57,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397757940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397757940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397757944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:57,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:57,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397757944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,012 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:58,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:58,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:58,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:58,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:58,015 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:58,015 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:58,015 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:58,015 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:58,016 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,016 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,019 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:58,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:58,020 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,020 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,020 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:58,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:58,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:58,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:58,021 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:58,021 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:58,021 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:58,021 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:58,021 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:58,021 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:58,021 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:58,021 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:58,021 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:58,021 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:58,022 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:58,022 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,022 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:58,022 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:58,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,024 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:58,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:58,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:58,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ccb8e14[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:58,025 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:58,026 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,027 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,027 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:58,027 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:58,027 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:58,027 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,027 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:58,028 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,028 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:58,029 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:58,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:58,029 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:58,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:58,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:58,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:58,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:58,031 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:58,031 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:58,031 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:58,031 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:58,031 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:58,033 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,033 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:58,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,033 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:58,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:58,034 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,035 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,038 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,038 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:58,038 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:58,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:58,038 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,038 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,038 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,038 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:58,038 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:58,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:58,038 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:58,039 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/d6597d5c65a44dda9863b8e032746c4f 2023-04-13 14:54:58,038 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:58,039 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:58,039 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:58,049 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/f2ed1ebd1b86494b8bb2a056d201ce0c 2023-04-13 14:54:58,139 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:58,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:58,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:58,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:58,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:58,142 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:58,142 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:58,142 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:58,142 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:58,143 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,143 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,144 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:58,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:58,144 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,145 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,145 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:58,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:58,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:58,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:58,146 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:58,146 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:58,146 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:58,146 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:58,148 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:58,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:58,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:58,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19de9047[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:58,149 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:58,149 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,151 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:58,151 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:58,151 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,151 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:58,151 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:58,153 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,153 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:58,153 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:58,153 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:58,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:58,155 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:58,156 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:58,156 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:58,156 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:58,156 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:58,157 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,157 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:58,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:58,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,157 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:58,158 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,159 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,161 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,161 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,161 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,161 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,161 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:58,161 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,161 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:58,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:58,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:58,161 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:58,161 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:58,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:58,161 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,162 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:58,162 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:58,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:58,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:58,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397758244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:58,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397758245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:58,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397758247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:58,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397758250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,262 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:58,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:58,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:58,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:58,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:58,265 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:58,265 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:58,265 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:58,266 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:58,266 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,266 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,267 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:58,267 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:58,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:58,267 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,267 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:58,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:58,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:58,269 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:58,269 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:58,269 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:58,269 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:58,269 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:58,269 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:58,269 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:58,269 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:58,269 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:58,269 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:58,269 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:58,269 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,269 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:58,269 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:58,272 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:58,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:58,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:58,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49f0db84[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:58,274 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:58,274 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,275 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,275 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:58,275 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:58,275 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,275 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:58,275 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:58,276 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,276 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:58,276 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:58,276 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:58,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:58,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:58,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:58,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:58,279 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:58,279 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:58,279 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:58,279 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:58,279 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:58,280 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,280 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:58,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:58,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,280 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:58,281 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:58,282 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:58,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:58,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:58,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:58,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:58,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:58,285 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:58,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:58,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:58,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:58,285 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:58,286 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,286 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:58,286 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,286 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,286 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,286 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,386 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:58,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:58,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:58,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:58,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:58,389 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:58,389 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:58,389 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:58,389 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:58,390 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,390 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:58,391 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:58,391 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,391 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:58,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:58,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:58,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:58,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:58,393 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:58,393 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:58,393 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:58,393 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:58,393 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:58,393 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:58,393 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:58,393 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:58,393 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:58,393 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,393 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:58,393 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:58,401 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,401 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:58,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:58,402 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,402 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:58,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:58,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:58,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:58,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:58,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24762408[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:58,403 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:58,403 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,405 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,405 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:58,405 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,405 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:58,405 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:58,406 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:58,407 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:58,407 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,407 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:58,407 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:58,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:58,409 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:58,409 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:58,409 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:58,410 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:58,410 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:58,411 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,411 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:58,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,411 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:58,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:58,412 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:58,413 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:58,415 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,415 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:58,415 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,415 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:58,416 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:58,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:58,415 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:58,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:58,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:58,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:58,416 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:58,416 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:58,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:58,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:58,459 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/bc3b444b5e8f4e878f4f76b47ea9914b 2023-04-13 14:54:58,463 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/d6597d5c65a44dda9863b8e032746c4f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d6597d5c65a44dda9863b8e032746c4f 2023-04-13 14:54:58,467 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d6597d5c65a44dda9863b8e032746c4f, entries=250, sequenceid=506, filesize=16.7 K 2023-04-13 14:54:58,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/f2ed1ebd1b86494b8bb2a056d201ce0c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f2ed1ebd1b86494b8bb2a056d201ce0c 2023-04-13 14:54:58,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f2ed1ebd1b86494b8bb2a056d201ce0c, entries=150, sequenceid=506, filesize=11.9 K 2023-04-13 14:54:58,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/bc3b444b5e8f4e878f4f76b47ea9914b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/bc3b444b5e8f4e878f4f76b47ea9914b 2023-04-13 14:54:58,476 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/bc3b444b5e8f4e878f4f76b47ea9914b, entries=150, sequenceid=506, filesize=11.9 K 2023-04-13 14:54:58,477 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for ccd2cd4c855e3a8c5393dd57d5cc737e in 847ms, sequenceid=506, compaction requested=true 2023-04-13 14:54:58,477 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:58,477 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:58,477 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:54:58,477 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:54:58,477 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:58,477 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:58,478 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54374 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:54:58,478 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:54:58,478 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/A is initiating minor compaction (all files) 2023-04-13 14:54:58,478 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/B is initiating minor compaction (all files) 2023-04-13 14:54:58,478 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/A in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:58,479 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/B in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:58,479 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/bb99fc423cb74a0788381bb5e1620d90, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/9761eb771b814f0a8a936eeebdce9ee6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/5de07c243e3d44df92458febe5ceaf52, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d6597d5c65a44dda9863b8e032746c4f] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=53.1 K 2023-04-13 14:54:58,479 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/d99bfcffcac9424bb2c80fd7d0ce3457, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0c88a3acb15046ffba3c1f2225c5af4c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/58aecf132b8848ad952b28a74f9b87e6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f2ed1ebd1b86494b8bb2a056d201ce0c] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=48.3 K 2023-04-13 14:54:58,479 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting bb99fc423cb74a0788381bb5e1620d90, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1681397694711 2023-04-13 14:54:58,479 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting d99bfcffcac9424bb2c80fd7d0ce3457, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1681397694711 2023-04-13 14:54:58,479 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 9761eb771b814f0a8a936eeebdce9ee6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681397694746 2023-04-13 14:54:58,480 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 5de07c243e3d44df92458febe5ceaf52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1681397695868 2023-04-13 14:54:58,480 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 0c88a3acb15046ffba3c1f2225c5af4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681397694746 2023-04-13 14:54:58,480 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting d6597d5c65a44dda9863b8e032746c4f, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1681397697007 2023-04-13 14:54:58,480 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 58aecf132b8848ad952b28a74f9b87e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1681397695868 2023-04-13 14:54:58,481 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting f2ed1ebd1b86494b8bb2a056d201ce0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1681397697012 2023-04-13 14:54:58,501 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#A#compaction#260 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:58,513 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#B#compaction#261 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:58,518 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:58,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:58,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:58,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:58,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:58,524 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:58,524 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:58,524 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:58,524 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:58,525 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,525 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,527 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:58,527 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:58,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:58,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:58,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:58,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:58,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:58,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:58,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:58,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:58,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:58,530 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:58,530 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:58,530 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 14:54:58,531 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:58,531 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:58,531 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:58,531 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:58,531 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:58,531 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:58,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:58,749 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:58,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397698784, max timestamp=1681397698791Num uniques 859; 2023-04-13 14:54:58,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397698752, max timestamp=1681397698791Num uniques 859; 2023-04-13 14:54:58,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681397698784, max timestamp=1681397698791Num uniques 859; 2023-04-13 14:54:58,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:58,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:54:58,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:54:58,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:58,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397758807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:58,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397758811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:58,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:58,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53214 deadline: 1681397758812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397758812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:58,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397758813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,817 DEBUG [Thread-1474] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8193 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:54:58,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:58,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397758913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:58,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397758916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:58,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397758916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:58,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397758917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:58,921 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/b01eeca06b8c482c927022c546ca901a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/b01eeca06b8c482c927022c546ca901a 2023-04-13 14:54:58,925 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/A of ccd2cd4c855e3a8c5393dd57d5cc737e into b01eeca06b8c482c927022c546ca901a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:58,925 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:58,925 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/A, priority=12, startTime=1681397698477; duration=0sec 2023-04-13 14:54:58,925 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:58,925 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:54:58,927 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:54:58,927 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/C is initiating minor compaction (all files) 2023-04-13 14:54:58,927 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/C in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:58,927 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/b1371e749f914c858db51b2bce0abae3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/4e066b3ddd4344b7a282ec4493c28079, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/dec2dd11cc3646f78d2fbf60c2ad3762, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/bc3b444b5e8f4e878f4f76b47ea9914b] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=48.3 K 2023-04-13 14:54:58,928 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting b1371e749f914c858db51b2bce0abae3, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1681397694711 2023-04-13 14:54:58,928 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 4e066b3ddd4344b7a282ec4493c28079, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681397694746 2023-04-13 14:54:58,929 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting dec2dd11cc3646f78d2fbf60c2ad3762, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1681397695868 2023-04-13 14:54:58,932 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting bc3b444b5e8f4e878f4f76b47ea9914b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1681397697012 2023-04-13 14:54:58,933 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/129a6568a90a4775b806b23975a4bb59 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/129a6568a90a4775b806b23975a4bb59 2023-04-13 14:54:58,941 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/B of ccd2cd4c855e3a8c5393dd57d5cc737e into 129a6568a90a4775b806b23975a4bb59(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:58,942 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:58,942 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/B, priority=12, startTime=1681397698477; duration=0sec 2023-04-13 14:54:58,942 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:58,944 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#C#compaction#263 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:58,950 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/6f66124b23164344b3a1c601fd7b3cef 2023-04-13 14:54:58,959 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/f771fbff733843188fa1f66d94bfe5c7 2023-04-13 14:54:59,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397759117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397759120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397759121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397759121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,354 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/e3aade70a8f140f997545efcc1e0d7b0 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e3aade70a8f140f997545efcc1e0d7b0 2023-04-13 14:54:59,358 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/C of ccd2cd4c855e3a8c5393dd57d5cc737e into e3aade70a8f140f997545efcc1e0d7b0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:59,358 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:59,358 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/C, priority=12, startTime=1681397698477; duration=0sec 2023-04-13 14:54:59,358 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:59,369 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/433aad42ef67468e8b738b3f53051a10 2023-04-13 14:54:59,373 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/6f66124b23164344b3a1c601fd7b3cef as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/6f66124b23164344b3a1c601fd7b3cef 2023-04-13 14:54:59,377 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/6f66124b23164344b3a1c601fd7b3cef, entries=150, sequenceid=515, filesize=11.9 K 2023-04-13 14:54:59,377 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/f771fbff733843188fa1f66d94bfe5c7 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f771fbff733843188fa1f66d94bfe5c7 2023-04-13 14:54:59,382 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f771fbff733843188fa1f66d94bfe5c7, entries=150, sequenceid=515, filesize=11.9 K 2023-04-13 14:54:59,383 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/433aad42ef67468e8b738b3f53051a10 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/433aad42ef67468e8b738b3f53051a10 2023-04-13 14:54:59,386 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/433aad42ef67468e8b738b3f53051a10, entries=150, sequenceid=515, filesize=11.9 K 2023-04-13 14:54:59,387 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=143.13 KB/146560 for ccd2cd4c855e3a8c5393dd57d5cc737e in 857ms, sequenceid=515, compaction requested=false 2023-04-13 14:54:59,387 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:59,387 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:59,389 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:59,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:59,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46db5bc[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:59,391 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:59,391 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,392 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:59,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:59,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:59,392 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,392 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:59,394 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:59,394 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:59,394 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:59,394 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:59,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:59,397 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:59,397 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:59,397 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:59,397 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:59,397 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:59,400 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,400 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:59,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,400 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:59,401 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:59,403 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,405 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:59,405 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:59,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:59,405 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,405 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,405 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,405 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,406 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,406 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,406 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:59,406 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:59,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:59,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:59,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:54:59,406 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:59,406 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:59,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:54:59,421 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=149.83 KB heapSize=425.92 KB 2023-04-13 14:54:59,422 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:59,422 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:54:59,422 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:59,422 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:54:59,422 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:59,422 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:54:59,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397759427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397759427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397759429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397759430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/fe1917e918c64e448b20eee744ec5011 2023-04-13 14:54:59,445 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/5ec032ddbb0a4ac2852c56769a63efed 2023-04-13 14:54:59,506 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:59,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:59,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:59,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:59,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:59,510 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:59,510 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:59,510 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:59,510 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:59,510 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,510 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,511 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:59,511 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:59,512 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,512 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:59,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:59,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:59,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:59,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:59,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:59,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:59,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:59,514 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:59,514 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:59,514 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:59,514 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:59,515 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:59,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:59,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:59,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69dc51de[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:59,517 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:59,517 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,519 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,519 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:59,519 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:59,519 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:59,519 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,520 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:59,521 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:59,521 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:59,521 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,521 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:59,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:59,523 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:59,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:59,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:59,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:59,523 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:59,525 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,525 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:59,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,525 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:59,525 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:59,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,526 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,528 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:59,528 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:59,528 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:59,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:59,528 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,529 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,529 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,529 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:59,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:59,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:59,529 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,529 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,529 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:59,529 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:59,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397759531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397759532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397759534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397759534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,629 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:59,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:59,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:59,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:59,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:59,632 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:59,632 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:59,632 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:59,632 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:59,633 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,633 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,635 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:59,635 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:59,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:59,635 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,635 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:59,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:59,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:59,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:59,636 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:59,636 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:59,636 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:59,637 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:59,637 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:59,637 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:59,637 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:59,637 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:59,637 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:59,637 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:59,637 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:59,637 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,637 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:59,637 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:59,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,638 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:59,639 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,639 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:59,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:59,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26bf8b9a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:59,640 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:59,641 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,642 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:59,642 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:59,642 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,642 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:59,642 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:59,648 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,648 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:59,648 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:59,648 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:59,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:59,651 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:59,651 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:59,651 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:59,651 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:59,651 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:59,652 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,652 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:59,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,652 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:59,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:59,653 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,654 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,657 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:59,657 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,657 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:59,657 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:59,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:59,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:59,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:59,657 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,657 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:59,657 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:59,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:59,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397759735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397759735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397759740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:54:59,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397759740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,757 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:59,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:59,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:59,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:59,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:59,760 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:59,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:59,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:59,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:59,761 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,761 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:59,765 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:59,766 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,766 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:59,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:59,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:59,767 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:59,767 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:59,767 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:54:59,767 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:59,769 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:59,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:59,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:59,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@337cbd21[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:59,770 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:59,770 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,771 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:59,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:59,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:59,772 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,772 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:59,773 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,773 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:59,773 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:59,773 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:54:59,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:54:59,775 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:54:59,775 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:54:59,775 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:54:59,775 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:54:59,775 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:54:59,783 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,783 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:59,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,783 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:59,783 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:59,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,784 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:54:59,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:54:59,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,787 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:59,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:59,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:59,787 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:54:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:59,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:54:59,787 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:54:59,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:54:59,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:54:59,861 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/691e2e5e789f467f8b36db1ecd1d66dd 2023-04-13 14:54:59,866 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/fe1917e918c64e448b20eee744ec5011 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/fe1917e918c64e448b20eee744ec5011 2023-04-13 14:54:59,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/fe1917e918c64e448b20eee744ec5011, entries=150, sequenceid=556, filesize=11.9 K 2023-04-13 14:54:59,870 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/5ec032ddbb0a4ac2852c56769a63efed as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5ec032ddbb0a4ac2852c56769a63efed 2023-04-13 14:54:59,874 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5ec032ddbb0a4ac2852c56769a63efed, entries=150, sequenceid=556, filesize=11.9 K 2023-04-13 14:54:59,874 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/691e2e5e789f467f8b36db1ecd1d66dd as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/691e2e5e789f467f8b36db1ecd1d66dd 2023-04-13 14:54:59,878 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/691e2e5e789f467f8b36db1ecd1d66dd, entries=150, sequenceid=556, filesize=11.9 K 2023-04-13 14:54:59,880 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~149.83 KB/153430, heapSize ~425.88 KB/436096, currentSize=33.54 KB/34350 for ccd2cd4c855e3a8c5393dd57d5cc737e in 459ms, sequenceid=556, compaction requested=true 2023-04-13 14:54:59,880 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:59,880 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:54:59,880 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:59,880 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:59,880 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:59,880 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:59,881 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:59,881 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:59,881 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/A is initiating minor compaction (all files) 2023-04-13 14:54:59,882 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/B is initiating minor compaction (all files) 2023-04-13 14:54:59,882 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/A in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:59,882 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/B in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:59,882 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/b01eeca06b8c482c927022c546ca901a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/6f66124b23164344b3a1c601fd7b3cef, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/fe1917e918c64e448b20eee744ec5011] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=36.6 K 2023-04-13 14:54:59,882 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/129a6568a90a4775b806b23975a4bb59, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f771fbff733843188fa1f66d94bfe5c7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5ec032ddbb0a4ac2852c56769a63efed] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=36.6 K 2023-04-13 14:54:59,882 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting b01eeca06b8c482c927022c546ca901a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1681397697012 2023-04-13 14:54:59,882 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 129a6568a90a4775b806b23975a4bb59, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1681397697012 2023-04-13 14:54:59,882 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 6f66124b23164344b3a1c601fd7b3cef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1681397697634 2023-04-13 14:54:59,883 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting f771fbff733843188fa1f66d94bfe5c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1681397697634 2023-04-13 14:54:59,883 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting fe1917e918c64e448b20eee744ec5011, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1681397698806 2023-04-13 14:54:59,883 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 5ec032ddbb0a4ac2852c56769a63efed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1681397698806 2023-04-13 14:54:59,888 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:54:59,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:54:59,889 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#B#compaction#269 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:59,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:54:59,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:54:59,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:54:59,891 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:54:59,891 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:59,891 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#A#compaction#270 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:59,891 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:54:59,891 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:54:59,895 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,895 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,896 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,896 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:54:59,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:54:59,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:54:59,896 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,897 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:54:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:54:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:54:59,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:54:59,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:54:59,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:54:59,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:54:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:54:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:54:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:54:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:54:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:54:59,898 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:59,898 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:54:59,899 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-13 14:54:59,899 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:54:59,899 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:59,900 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:54:59,900 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:59,900 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:54:59,900 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:54:59,901 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/73983eb2cf304968b6b0d9992a5d3b37 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/73983eb2cf304968b6b0d9992a5d3b37 2023-04-13 14:54:59,906 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/B of ccd2cd4c855e3a8c5393dd57d5cc737e into 73983eb2cf304968b6b0d9992a5d3b37(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:54:59,906 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:59,906 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/B, priority=13, startTime=1681397699880; duration=0sec 2023-04-13 14:54:59,906 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:54:59,906 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:54:59,910 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/180b21d03dca4c1a9c58ea15e09ca1a7 2023-04-13 14:54:59,910 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:54:59,910 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/C is initiating minor compaction (all files) 2023-04-13 14:54:59,910 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/C in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:59,910 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e3aade70a8f140f997545efcc1e0d7b0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/433aad42ef67468e8b738b3f53051a10, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/691e2e5e789f467f8b36db1ecd1d66dd] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=36.6 K 2023-04-13 14:54:59,911 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting e3aade70a8f140f997545efcc1e0d7b0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1681397697012 2023-04-13 14:54:59,911 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 433aad42ef67468e8b738b3f53051a10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1681397697634 2023-04-13 14:54:59,912 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 691e2e5e789f467f8b36db1ecd1d66dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1681397698806 2023-04-13 14:54:59,918 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#C#compaction#273 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:54:59,921 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/06e33d2680f341f595fa803ef38446bf 2023-04-13 14:54:59,962 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/9d93ce38cc354150979a384f1980f45b 2023-04-13 14:54:59,968 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/180b21d03dca4c1a9c58ea15e09ca1a7 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/180b21d03dca4c1a9c58ea15e09ca1a7 2023-04-13 14:54:59,972 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/180b21d03dca4c1a9c58ea15e09ca1a7, entries=100, sequenceid=564, filesize=9.5 K 2023-04-13 14:54:59,973 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/06e33d2680f341f595fa803ef38446bf as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/06e33d2680f341f595fa803ef38446bf 2023-04-13 14:54:59,977 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/06e33d2680f341f595fa803ef38446bf, entries=100, sequenceid=564, filesize=9.5 K 2023-04-13 14:54:59,979 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/9d93ce38cc354150979a384f1980f45b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/9d93ce38cc354150979a384f1980f45b 2023-04-13 14:54:59,985 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/9d93ce38cc354150979a384f1980f45b, entries=100, sequenceid=564, filesize=9.5 K 2023-04-13 14:54:59,985 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for ccd2cd4c855e3a8c5393dd57d5cc737e in 87ms, sequenceid=564, compaction requested=false 2023-04-13 14:54:59,985 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:54:59,985 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:54:59,985 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:54:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:54:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,988 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:54:59,988 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:54:59,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,990 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:59,990 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,990 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:54:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:54:59,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:54:59,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a553ea7[Count = 0] remaining members to acquire global barrier 2023-04-13 14:54:59,993 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:54:59,993 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,995 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:54:59,995 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:54:59,995 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,995 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:54:59,995 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:54:59,995 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:54:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:54:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:54:59,997 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:54:59,997 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:54:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:54:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:54:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:54:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:54:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:54:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:00,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:00,001 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:00,001 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:00,001 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:00,001 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:00,001 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:00,003 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,003 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,003 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,004 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,010 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:00,010 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:00,010 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:00,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:00,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,011 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:00,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:00,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,011 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:00,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,011 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:00,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:55:00,047 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:55:00,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:55:00,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:00,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:55:00,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:00,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:55:00,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:00,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397700061, max timestamp=1681397700069Num uniques 859; 2023-04-13 14:55:00,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397700061, max timestamp=1681397700069Num uniques 859; 2023-04-13 14:55:00,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:00,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397700061, max timestamp=1681397700069Num uniques 859; 2023-04-13 14:55:00,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:00,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:00,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397760085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397760088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397760090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397760090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,111 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:00,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:00,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:00,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:00,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:00,115 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:00,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:00,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:00,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:00,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:00,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:00,117 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,118 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,118 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:00,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:00,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:00,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:00,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:00,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:00,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:00,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:00,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:00,119 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,119 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:55:00,120 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:55:00,120 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:00,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:00,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:00,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:00,122 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:00,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@518cc692[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:00,123 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:00,124 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,125 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,125 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:00,125 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:00,125 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:00,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:00,126 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:00,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:00,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:00,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:00,129 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:00,129 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:00,129 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:00,129 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:00,129 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:00,136 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,136 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,136 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,136 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,138 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,140 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,140 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:00,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:00,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:00,140 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:00,140 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:00,141 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:00,141 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:00,140 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,141 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,141 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,141 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,141 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:00,141 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397760191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397760192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397760194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397760194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,241 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:00,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:00,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:00,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:00,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:00,244 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:00,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:00,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:00,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:00,245 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,245 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,246 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:00,246 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:00,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:00,246 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,246 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:00,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:00,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:00,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:00,247 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:00,247 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:00,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:55:00,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:55:00,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:00,250 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,251 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,251 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,251 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:00,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:00,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bb56462[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:00,252 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:00,253 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,253 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,254 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:00,254 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:00,254 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:00,254 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,254 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:00,255 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:00,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:00,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:00,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:00,257 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:00,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:00,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:00,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:00,258 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:00,259 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,259 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,259 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,260 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,261 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:00,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,264 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:00,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:00,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:00,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:00,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:00,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,265 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:00,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,265 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:00,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:00,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,305 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/1b9db0f013d440f48289e2ddafc49f8d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/1b9db0f013d440f48289e2ddafc49f8d 2023-04-13 14:55:00,310 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/A of ccd2cd4c855e3a8c5393dd57d5cc737e into 1b9db0f013d440f48289e2ddafc49f8d(size=12.9 K), total size for store is 22.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:00,310 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:55:00,310 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/A, priority=13, startTime=1681397699880; duration=0sec 2023-04-13 14:55:00,310 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:00,341 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/19ba66e52e314ffd8d71d5c87560dfbb as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/19ba66e52e314ffd8d71d5c87560dfbb 2023-04-13 14:55:00,345 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/C of ccd2cd4c855e3a8c5393dd57d5cc737e into 19ba66e52e314ffd8d71d5c87560dfbb(size=12.9 K), total size for store is 22.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:00,345 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:55:00,345 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/C, priority=13, startTime=1681397699880; duration=0sec 2023-04-13 14:55:00,346 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:00,365 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:00,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:00,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:00,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:00,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:00,368 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:00,368 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:00,368 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:00,368 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:00,368 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,368 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,370 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:00,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:00,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:00,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:00,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:00,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:00,371 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:00,371 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:00,371 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:00,372 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:00,372 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:00,372 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,372 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:55:00,372 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:55:00,372 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,372 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:00,372 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:00,372 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,372 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:00,372 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:00,373 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,373 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,374 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,374 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:00,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d764580[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:00,375 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:00,375 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,376 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:00,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:00,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:00,376 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,376 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:00,379 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,379 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,379 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:00,379 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:00,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:00,382 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:00,382 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:00,382 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:00,382 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:00,382 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:00,383 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,383 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,383 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,384 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,385 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,387 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:00,387 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,387 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:00,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:00,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:00,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:00,387 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:00,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,387 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,388 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:00,388 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:00,388 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,388 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,388 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,388 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,388 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397760394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397760396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397760397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397760399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/62d86574201447cfadb46a9cb1ca0c89 2023-04-13 14:55:00,488 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:00,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:00,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:00,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:00,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:00,491 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:00,491 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:00,491 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:00,491 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:00,491 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,491 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,493 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:00,493 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:00,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:00,493 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,493 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:00,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:00,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:00,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:00,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:00,494 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,494 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:55:00,494 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:55:00,494 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:00,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,497 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,497 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,497 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:00,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:00,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e25e88f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:00,498 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:00,498 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,500 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,500 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:00,500 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:00,500 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:00,500 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,500 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:00,501 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,501 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:00,501 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:00,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,501 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:00,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:00,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:00,503 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:00,503 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:00,503 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:00,503 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:00,503 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:00,505 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,505 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,505 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,505 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,506 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,511 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:00,511 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:00,511 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:00,512 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:00,512 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,512 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:00,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:00,512 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,512 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,512 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,512 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,512 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:00,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:00,512 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:00,612 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:00,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:00,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:00,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:00,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:00,616 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:00,616 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:00,616 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:00,616 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:00,616 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,616 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,618 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,618 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:00,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:00,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:00,618 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,618 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:00,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:00,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:00,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:00,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:00,619 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:00,619 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:00,619 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:00,619 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:00,619 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:00,620 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,620 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:55:00,620 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:55:00,620 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,620 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:00,620 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:00,620 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,620 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:00,620 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:00,621 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,621 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,621 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,621 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:00,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:00,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@192a138f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:00,623 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:00,623 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,625 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,625 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:00,625 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,625 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:00,625 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:00,625 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:00,627 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,627 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,627 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:00,627 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:00,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:00,636 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:00,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:00,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:00,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:00,636 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:00,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,638 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,639 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:00,642 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:00,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:00,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:00,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:00,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:00,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,642 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:00,642 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:00,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397760698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397760700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397760701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:00,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397760703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,742 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:00,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:00,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:00,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:00,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:00,746 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:00,746 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:00,746 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:00,746 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:00,746 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,746 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,748 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:00,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:00,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:00,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:00,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:00,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:00,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:00,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:00,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:00,750 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,750 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:55:00,750 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:55:00,750 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:00,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:00,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:00,750 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:00,751 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,752 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,752 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:00,753 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:00,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@391cf2bc[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:00,753 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,754 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,754 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:00,755 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:00,755 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:00,755 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,755 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:00,756 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:00,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:00,758 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:00,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:00,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:00,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:00,758 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:00,761 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,761 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,761 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,761 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,762 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,764 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,764 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:00,764 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:00,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:00,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:00,764 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:00,765 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:00,764 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:00,764 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,765 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:00,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,865 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:00,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:00,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:00,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:00,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:00,868 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:00,868 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:00,868 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:00,868 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:00,868 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,868 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,870 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:00,870 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:00,870 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,870 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:00,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:00,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:00,872 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:00,872 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:00,872 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,872 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:55:00,872 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:55:00,872 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,872 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:00,872 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:00,872 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,872 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:00,872 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:00,873 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,874 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,874 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,874 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:00,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56a9fa5a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:00,875 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:00,875 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,876 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/0e492ed27d5d492aa31af3fd1a0d4d65 2023-04-13 14:55:00,877 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:00,878 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,878 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:00,879 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:00,879 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:00,879 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:00,879 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:00,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:00,882 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:00,883 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:00,883 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:00,883 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:00,883 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:00,885 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,885 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,885 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:00,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,886 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,887 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/642f1f208960451ebe9342141b7cd980 2023-04-13 14:55:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,888 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:00,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,891 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,891 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,891 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:00,891 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:00,891 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,891 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:00,891 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,891 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:00,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:00,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:00,891 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:00,891 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:00,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:00,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:00,892 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:00,892 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:00,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:00,892 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/62d86574201447cfadb46a9cb1ca0c89 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/62d86574201447cfadb46a9cb1ca0c89 2023-04-13 14:55:00,896 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/62d86574201447cfadb46a9cb1ca0c89, entries=150, sequenceid=575, filesize=11.9 K 2023-04-13 14:55:00,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/0e492ed27d5d492aa31af3fd1a0d4d65 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0e492ed27d5d492aa31af3fd1a0d4d65 2023-04-13 14:55:00,900 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0e492ed27d5d492aa31af3fd1a0d4d65, entries=100, sequenceid=575, filesize=9.5 K 2023-04-13 14:55:00,901 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/642f1f208960451ebe9342141b7cd980 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/642f1f208960451ebe9342141b7cd980 2023-04-13 14:55:00,904 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/642f1f208960451ebe9342141b7cd980, entries=100, sequenceid=575, filesize=9.5 K 2023-04-13 14:55:00,905 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ccd2cd4c855e3a8c5393dd57d5cc737e in 858ms, sequenceid=575, compaction requested=true 2023-04-13 14:55:00,905 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:55:00,905 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:00,905 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:00,905 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:00,905 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:00,905 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:00,906 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35098 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:00,906 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 32658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:00,906 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/A is initiating minor compaction (all files) 2023-04-13 14:55:00,906 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/B is initiating minor compaction (all files) 2023-04-13 14:55:00,906 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/A in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,906 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/B in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:00,906 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/1b9db0f013d440f48289e2ddafc49f8d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/180b21d03dca4c1a9c58ea15e09ca1a7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/62d86574201447cfadb46a9cb1ca0c89] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=34.3 K 2023-04-13 14:55:00,906 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/73983eb2cf304968b6b0d9992a5d3b37, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/06e33d2680f341f595fa803ef38446bf, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0e492ed27d5d492aa31af3fd1a0d4d65] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=31.9 K 2023-04-13 14:55:00,907 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 1b9db0f013d440f48289e2ddafc49f8d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1681397698806 2023-04-13 14:55:00,907 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 73983eb2cf304968b6b0d9992a5d3b37, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1681397698806 2023-04-13 14:55:00,907 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 180b21d03dca4c1a9c58ea15e09ca1a7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1681397699428 2023-04-13 14:55:00,907 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 06e33d2680f341f595fa803ef38446bf, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1681397699428 2023-04-13 14:55:00,907 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 62d86574201447cfadb46a9cb1ca0c89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681397700043 2023-04-13 14:55:00,908 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 0e492ed27d5d492aa31af3fd1a0d4d65, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681397700043 2023-04-13 14:55:00,921 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#B#compaction#279 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:00,921 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#A#compaction#278 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:00,992 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:00,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:00,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:00,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:00,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:00,995 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:00,995 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:00,995 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:00,995 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:00,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:00,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:01,004 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,004 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:01,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:01,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:01,004 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,004 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:01,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:01,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:01,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:01,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:01,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:01,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:01,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:01,005 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:01,006 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:01,006 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:01,006 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:01,006 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:01,006 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:01,006 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:55:01,006 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-13 14:55:01,006 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:55:01,006 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:01,006 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:55:01,006 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:01,006 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:55:01,006 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:01,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:55:01,205 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:55:01,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397761211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397761212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397761214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397761214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397761315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397761315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397761317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397761317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,338 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/ffb009d2650a460aab342da4916b77bf as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/ffb009d2650a460aab342da4916b77bf 2023-04-13 14:55:01,339 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/906587c272a34135af265c07d51ab6da as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/906587c272a34135af265c07d51ab6da 2023-04-13 14:55:01,342 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/B of ccd2cd4c855e3a8c5393dd57d5cc737e into ffb009d2650a460aab342da4916b77bf(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:01,342 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:55:01,342 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/B, priority=13, startTime=1681397700905; duration=0sec 2023-04-13 14:55:01,342 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:01,342 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:01,344 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 32658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:01,344 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/C is initiating minor compaction (all files) 2023-04-13 14:55:01,345 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/C in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:01,345 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/A of ccd2cd4c855e3a8c5393dd57d5cc737e into 906587c272a34135af265c07d51ab6da(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:01,345 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/19ba66e52e314ffd8d71d5c87560dfbb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/9d93ce38cc354150979a384f1980f45b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/642f1f208960451ebe9342141b7cd980] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=31.9 K 2023-04-13 14:55:01,345 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:55:01,345 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/A, priority=13, startTime=1681397700905; duration=0sec 2023-04-13 14:55:01,345 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:01,345 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 19ba66e52e314ffd8d71d5c87560dfbb, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1681397698806 2023-04-13 14:55:01,345 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 9d93ce38cc354150979a384f1980f45b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1681397699428 2023-04-13 14:55:01,346 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 642f1f208960451ebe9342141b7cd980, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681397700043 2023-04-13 14:55:01,351 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#C#compaction#281 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:01,413 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/11bdec9adab54cc6891f2b5905fc243e 2023-04-13 14:55:01,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397761516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397761516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397761520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397761521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,765 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/e5ce6a3dc7024b69a84d7be65fb9af2e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e5ce6a3dc7024b69a84d7be65fb9af2e 2023-04-13 14:55:01,769 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/C of ccd2cd4c855e3a8c5393dd57d5cc737e into e5ce6a3dc7024b69a84d7be65fb9af2e(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:01,770 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:55:01,770 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/C, priority=13, startTime=1681397700905; duration=0sec 2023-04-13 14:55:01,770 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:01,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397761820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397761821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397761822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,825 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/392021fea6e74c948f45ce58d2e359c0 2023-04-13 14:55:01,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:01,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397761824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,838 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/6b61d592f58f4488a2fd6141ccfa5ea5 2023-04-13 14:55:01,841 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/11bdec9adab54cc6891f2b5905fc243e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/11bdec9adab54cc6891f2b5905fc243e 2023-04-13 14:55:01,844 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/11bdec9adab54cc6891f2b5905fc243e, entries=150, sequenceid=613, filesize=11.9 K 2023-04-13 14:55:01,845 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/392021fea6e74c948f45ce58d2e359c0 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/392021fea6e74c948f45ce58d2e359c0 2023-04-13 14:55:01,848 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/392021fea6e74c948f45ce58d2e359c0, entries=150, sequenceid=613, filesize=11.9 K 2023-04-13 14:55:01,849 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/6b61d592f58f4488a2fd6141ccfa5ea5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/6b61d592f58f4488a2fd6141ccfa5ea5 2023-04-13 14:55:01,856 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/6b61d592f58f4488a2fd6141ccfa5ea5, entries=150, sequenceid=613, filesize=11.9 K 2023-04-13 14:55:01,857 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=53.67 KB/54960 for ccd2cd4c855e3a8c5393dd57d5cc737e in 851ms, sequenceid=613, compaction requested=false 2023-04-13 14:55:01,857 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:55:01,857 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:01,858 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:01,858 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:01,858 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,858 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:01,858 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:01,860 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:01,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:01,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:01,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:01,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:01,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:01,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:01,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57490190[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:01,862 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:01,862 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:01,864 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:01,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:01,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:01,864 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:01,864 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:01,864 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:01,865 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,865 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:01,867 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:01,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:01,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:01,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:01,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:01,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:01,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:01,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:01,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:01,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:01,870 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:01,870 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:01,870 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:01,870 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:01,870 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:01,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:01,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:01,872 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:01,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:01,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:01,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:01,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:01,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:01,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:01,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:01,872 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:01,874 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,876 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,876 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:01,876 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:01,876 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:01,876 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:01,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:01,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:01,877 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,877 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:01,877 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:01,877 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:01,877 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:01,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:01,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:01,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:01,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:01,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:01,877 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:01,877 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:01,977 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:01,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:01,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:01,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:01,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:01,981 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:01,981 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:01,981 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:01,981 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:01,981 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:01,981 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:01,983 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:01,983 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:01,983 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:01,983 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:01,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:01,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:01,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:01,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:01,984 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:01,984 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:01,984 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:01,984 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:01,984 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:01,985 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:01,985 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:55:01,985 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 14:55:01,985 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:55:01,985 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:01,985 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:55:01,985 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:01,985 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:55:01,985 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:02,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:55:02,325 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:55:02,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397702337, max timestamp=1681397702343Num uniques 859; 2023-04-13 14:55:02,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:02,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397702337, max timestamp=1681397702343Num uniques 859; 2023-04-13 14:55:02,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:02,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397702337, max timestamp=1681397702343Num uniques 859; 2023-04-13 14:55:02,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:02,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397762359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397762360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397762360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397762363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,395 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/6eba45465b3a44609dac4bdcd87da2e7 2023-04-13 14:55:02,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397762464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397762464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397762464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397762466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397762666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397762666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397762667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397762670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,806 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/0495a2242525416796b510e3267b8b66 2023-04-13 14:55:02,815 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/fdeef05216aa4fcd8591eaa5b32a8114 2023-04-13 14:55:02,819 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/6eba45465b3a44609dac4bdcd87da2e7 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/6eba45465b3a44609dac4bdcd87da2e7 2023-04-13 14:55:02,823 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/6eba45465b3a44609dac4bdcd87da2e7, entries=150, sequenceid=627, filesize=11.9 K 2023-04-13 14:55:02,824 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/0495a2242525416796b510e3267b8b66 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0495a2242525416796b510e3267b8b66 2023-04-13 14:55:02,827 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0495a2242525416796b510e3267b8b66, entries=150, sequenceid=627, filesize=11.9 K 2023-04-13 14:55:02,827 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/fdeef05216aa4fcd8591eaa5b32a8114 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/fdeef05216aa4fcd8591eaa5b32a8114 2023-04-13 14:55:02,831 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/fdeef05216aa4fcd8591eaa5b32a8114, entries=150, sequenceid=627, filesize=11.9 K 2023-04-13 14:55:02,832 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ccd2cd4c855e3a8c5393dd57d5cc737e in 847ms, sequenceid=627, compaction requested=true 2023-04-13 14:55:02,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:55:02,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:02,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:02,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:02,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:02,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:02,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:02,835 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:02,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:02,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:02,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:02,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb6c69d[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:02,837 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:02,837 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:02,838 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:02,838 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:02,838 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:02,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:02,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:02,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:02,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:02,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:02,840 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:02,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:02,842 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:02,842 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:02,842 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:02,842 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:02,843 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:02,844 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:02,844 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:02,844 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:02,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:02,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:02,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,846 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:02,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:02,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:02,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:02,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:02,849 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:02,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:02,849 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:02,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:02,849 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:02,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:02,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:02,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:02,850 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:02,949 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:02,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:02,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:02,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:02,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:02,952 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:02,952 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:02,952 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:02,953 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:02,953 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:02,953 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:02,954 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:02,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:02,954 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:02,955 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,955 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:02,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:02,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:02,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:02,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:02,956 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:02,956 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:02,956 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:02,956 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:02,956 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:02,956 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:02,956 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. started... 2023-04-13 14:55:02,956 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-13 14:55:02,956 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:55:02,956 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:02,956 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:55:02,956 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:02,956 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:55:02,956 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:02,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:55:02,969 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. as already flushing 2023-04-13 14:55:02,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397762977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397762978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397762978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:02,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:02,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397762979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:03,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:03,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397763081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:03,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:03,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397763084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:03,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:03,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397763084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:03,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:03,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397763084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:03,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:03,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397763285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:03,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:03,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397763287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:03,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:03,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397763287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:03,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:03,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397763288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:03,364 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=662 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/30f5efe41761461684a742faf496127c 2023-04-13 14:55:03,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:03,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397763587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:03,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:03,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397763590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:03,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:03,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397763590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:03,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:03,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397763591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:03,774 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=662 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/ee33825ae3564862b8d92071a2ef38d6 2023-04-13 14:55:03,813 DEBUG [Thread-1481] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dea0f9b to 127.0.0.1:62626 2023-04-13 14:55:03,814 DEBUG [Thread-1481] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:03,814 DEBUG [Thread-1479] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x753156b2 to 127.0.0.1:62626 2023-04-13 14:55:03,815 DEBUG [Thread-1479] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:03,815 DEBUG [Thread-1477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x104dbf4e to 127.0.0.1:62626 2023-04-13 14:55:03,815 DEBUG [Thread-1477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:03,815 DEBUG [Thread-1483] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x434e0eab to 127.0.0.1:62626 2023-04-13 14:55:03,815 DEBUG [Thread-1483] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:03,817 DEBUG [Thread-1485] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c43dddd to 127.0.0.1:62626 2023-04-13 14:55:03,817 DEBUG [Thread-1485] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:04,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:04,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53236 deadline: 1681397764093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:04,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53274 deadline: 1681397764093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:04,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1681397764095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:04,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53292 deadline: 1681397764097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,184 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=662 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/240c5ab302334d5184b4a64d70da8820 2023-04-13 14:55:04,187 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/30f5efe41761461684a742faf496127c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/30f5efe41761461684a742faf496127c 2023-04-13 14:55:04,190 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/30f5efe41761461684a742faf496127c, entries=150, sequenceid=662, filesize=11.9 K 2023-04-13 14:55:04,191 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/ee33825ae3564862b8d92071a2ef38d6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/ee33825ae3564862b8d92071a2ef38d6 2023-04-13 14:55:04,195 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/ee33825ae3564862b8d92071a2ef38d6, entries=150, sequenceid=662, filesize=11.9 K 2023-04-13 14:55:04,195 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/240c5ab302334d5184b4a64d70da8820 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/240c5ab302334d5184b4a64d70da8820 2023-04-13 14:55:04,198 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/240c5ab302334d5184b4a64d70da8820, entries=150, sequenceid=662, filesize=11.9 K 2023-04-13 14:55:04,199 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for ccd2cd4c855e3a8c5393dd57d5cc737e in 1243ms, sequenceid=662, compaction requested=true 2023-04-13 14:55:04,199 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:55:04,199 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:04,199 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:04,199 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:04,199 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,199 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:04,199 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:04,200 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,200 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:04,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:04,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:04,201 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:04,201 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:04,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:04,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:04,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:04,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:04,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40614ee5[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:04,202 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:04,202 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:04,204 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:04,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:04,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:04,204 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,204 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:04,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:04,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:04,206 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:04,206 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,206 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:04,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:04,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:04,206 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:04,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:04,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:04,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:04,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:04,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:04,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:04,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:04,208 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:04,208 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:04,208 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:04,208 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:04,208 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:04,209 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:04,209 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:04,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:04,209 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:04,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:04,210 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:04,211 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:04,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:04,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,212 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,212 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:04,212 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:04,212 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:04,212 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:04,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:04,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:04,212 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:04,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:04,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:04,213 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:04,213 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:04,213 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:04,213 DEBUG [Thread-1476] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:04,213 DEBUG [Thread-1476] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:04,213 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:04,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:04,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:04,313 DEBUG [Thread-1476] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:04,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:05,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:55:05,101 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 14:55:05,101 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:55:05,101 DEBUG [Thread-1468] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70043cf0 to 127.0.0.1:62626 2023-04-13 14:55:05,101 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:05,101 DEBUG [Thread-1468] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:05,101 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:55:05,101 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:05,101 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:55:05,101 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:05,103 DEBUG [Thread-1470] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b7fed0 to 127.0.0.1:62626 2023-04-13 14:55:05,103 DEBUG [Thread-1470] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:05,105 DEBUG [Thread-1472] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f4ad99d to 127.0.0.1:62626 2023-04-13 14:55:05,105 DEBUG [Thread-1472] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:05,106 DEBUG [Thread-1466] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ed1dde9 to 127.0.0.1:62626 2023-04-13 14:55:05,106 DEBUG [Thread-1466] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:05,509 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/540c644dbcce40748187d1e1aa86bf28 2023-04-13 14:55:05,918 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/45032a255fb94beaa5859208b3bf3b2e 2023-04-13 14:55:06,326 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/e4dce7d86ab6466799adf9b7b599ec40 2023-04-13 14:55:06,330 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/540c644dbcce40748187d1e1aa86bf28 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/540c644dbcce40748187d1e1aa86bf28 2023-04-13 14:55:06,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/540c644dbcce40748187d1e1aa86bf28, entries=150, sequenceid=675, filesize=11.9 K 2023-04-13 14:55:06,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/45032a255fb94beaa5859208b3bf3b2e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/45032a255fb94beaa5859208b3bf3b2e 2023-04-13 14:55:06,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/45032a255fb94beaa5859208b3bf3b2e, entries=150, sequenceid=675, filesize=11.9 K 2023-04-13 14:55:06,337 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/e4dce7d86ab6466799adf9b7b599ec40 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e4dce7d86ab6466799adf9b7b599ec40 2023-04-13 14:55:06,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e4dce7d86ab6466799adf9b7b599ec40, entries=150, sequenceid=675, filesize=11.9 K 2023-04-13 14:55:06,340 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=20.13 KB/20610 for ccd2cd4c855e3a8c5393dd57d5cc737e in 1239ms, sequenceid=675, compaction requested=true 2023-04-13 14:55:06,340 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:55:06,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:06,340 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 14:55:06,340 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 14:55:06,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:06,341 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:06,342 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62020 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 14:55:06,342 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/A is initiating minor compaction (all files) 2023-04-13 14:55:06,342 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62020 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 14:55:06,342 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/A in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:06,342 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/B is initiating minor compaction (all files) 2023-04-13 14:55:06,342 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/906587c272a34135af265c07d51ab6da, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/11bdec9adab54cc6891f2b5905fc243e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/6eba45465b3a44609dac4bdcd87da2e7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/30f5efe41761461684a742faf496127c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/540c644dbcce40748187d1e1aa86bf28] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=60.6 K 2023-04-13 14:55:06,342 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/B in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:06,342 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/ffb009d2650a460aab342da4916b77bf, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/392021fea6e74c948f45ce58d2e359c0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0495a2242525416796b510e3267b8b66, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/ee33825ae3564862b8d92071a2ef38d6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/45032a255fb94beaa5859208b3bf3b2e] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=60.6 K 2023-04-13 14:55:06,342 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 906587c272a34135af265c07d51ab6da, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681397699428 2023-04-13 14:55:06,342 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting ffb009d2650a460aab342da4916b77bf, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681397699428 2023-04-13 14:55:06,342 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 11bdec9adab54cc6891f2b5905fc243e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=613, earliestPutTs=1681397700084 2023-04-13 14:55:06,343 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 392021fea6e74c948f45ce58d2e359c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=613, earliestPutTs=1681397700084 2023-04-13 14:55:06,343 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 6eba45465b3a44609dac4bdcd87da2e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1681397701209 2023-04-13 14:55:06,343 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 0495a2242525416796b510e3267b8b66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1681397701209 2023-04-13 14:55:06,343 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 30f5efe41761461684a742faf496127c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=662, earliestPutTs=1681397702358 2023-04-13 14:55:06,343 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting ee33825ae3564862b8d92071a2ef38d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=662, earliestPutTs=1681397702358 2023-04-13 14:55:06,343 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 540c644dbcce40748187d1e1aa86bf28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1681397702968 2023-04-13 14:55:06,343 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 45032a255fb94beaa5859208b3bf3b2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1681397702968 2023-04-13 14:55:06,351 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#A#compaction#293 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:06,353 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#B#compaction#294 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:06,759 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/9218535f10d84a47997dda11a8d4e725 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/9218535f10d84a47997dda11a8d4e725 2023-04-13 14:55:06,759 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/cb9efb3949ea426f82ec1a1d712c2024 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/cb9efb3949ea426f82ec1a1d712c2024 2023-04-13 14:55:06,763 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/A of ccd2cd4c855e3a8c5393dd57d5cc737e into 9218535f10d84a47997dda11a8d4e725(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:06,763 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:55:06,763 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/A, priority=11, startTime=1681397706340; duration=0sec 2023-04-13 14:55:06,763 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:06,763 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 14:55:06,763 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/B of ccd2cd4c855e3a8c5393dd57d5cc737e into cb9efb3949ea426f82ec1a1d712c2024(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:06,763 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:55:06,763 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/B, priority=11, startTime=1681397706340; duration=0sec 2023-04-13 14:55:06,763 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:06,764 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62020 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 14:55:06,764 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): ccd2cd4c855e3a8c5393dd57d5cc737e/C is initiating minor compaction (all files) 2023-04-13 14:55:06,764 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ccd2cd4c855e3a8c5393dd57d5cc737e/C in TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:06,764 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e5ce6a3dc7024b69a84d7be65fb9af2e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/6b61d592f58f4488a2fd6141ccfa5ea5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/fdeef05216aa4fcd8591eaa5b32a8114, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/240c5ab302334d5184b4a64d70da8820, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e4dce7d86ab6466799adf9b7b599ec40] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp, totalSize=60.6 K 2023-04-13 14:55:06,765 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting e5ce6a3dc7024b69a84d7be65fb9af2e, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681397699428 2023-04-13 14:55:06,765 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 6b61d592f58f4488a2fd6141ccfa5ea5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=613, earliestPutTs=1681397700084 2023-04-13 14:55:06,765 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting fdeef05216aa4fcd8591eaa5b32a8114, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1681397701209 2023-04-13 14:55:06,765 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 240c5ab302334d5184b4a64d70da8820, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=662, earliestPutTs=1681397702358 2023-04-13 14:55:06,765 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting e4dce7d86ab6466799adf9b7b599ec40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1681397702968 2023-04-13 14:55:06,773 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): ccd2cd4c855e3a8c5393dd57d5cc737e#C#compaction#295 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:07,180 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/5db4ccbb020f4b2da619c8ae21476d8b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/5db4ccbb020f4b2da619c8ae21476d8b 2023-04-13 14:55:07,184 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ccd2cd4c855e3a8c5393dd57d5cc737e/C of ccd2cd4c855e3a8c5393dd57d5cc737e into 5db4ccbb020f4b2da619c8ae21476d8b(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:07,184 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:55:07,184 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e., storeName=ccd2cd4c855e3a8c5393dd57d5cc737e/C, priority=11, startTime=1681397706341; duration=0sec 2023-04-13 14:55:07,185 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:08,881 DEBUG [Thread-1474] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x330b77da to 127.0.0.1:62626 2023-04-13 14:55:08,881 DEBUG [Thread-1474] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:08,881 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 14:55:08,881 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 136 2023-04-13 14:55:08,881 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 110 2023-04-13 14:55:08,881 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 112 2023-04-13 14:55:08,881 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 117 2023-04-13 14:55:08,881 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 92 2023-04-13 14:55:08,881 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 14:55:08,881 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(398): read 5196 2023-04-13 14:55:08,882 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(398): read 4993 2023-04-13 14:55:08,882 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(398): read 5351 2023-04-13 14:55:08,882 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(398): read 5187 2023-04-13 14:55:08,882 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(398): read 4996 2023-04-13 14:55:08,882 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 14:55:08,882 INFO [Listener at localhost/37101] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 14:55:08,882 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x112921f2 to 127.0.0.1:62626 2023-04-13 14:55:08,882 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:08,882 INFO [Listener at localhost/37101] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 14:55:08,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 14:55:08,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:08,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 14:55:08,886 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397708886"}]},"ts":"1681397708886"} 2023-04-13 14:55:08,887 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 14:55:08,889 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 14:55:08,889 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ccd2cd4c855e3a8c5393dd57d5cc737e, UNASSIGN}] 2023-04-13 14:55:08,890 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ccd2cd4c855e3a8c5393dd57d5cc737e, UNASSIGN 2023-04-13 14:55:08,891 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=ccd2cd4c855e3a8c5393dd57d5cc737e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:08,891 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397708891"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397708891"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397708891"}]},"ts":"1681397708891"} 2023-04-13 14:55:08,892 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:55:08,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 14:55:09,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:55:09,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ccd2cd4c855e3a8c5393dd57d5cc737e, disabling compactions & flushes 2023-04-13 14:55:09,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:09,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:09,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. after waiting 0 ms 2023-04-13 14:55:09,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:09,044 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing ccd2cd4c855e3a8c5393dd57d5cc737e 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 14:55:09,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=A 2023-04-13 14:55:09,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:09,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=B 2023-04-13 14:55:09,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:09,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ccd2cd4c855e3a8c5393dd57d5cc737e, store=C 2023-04-13 14:55:09,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:09,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 14:55:09,450 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/194a7f2eed6e48bd8337a6f4b9b02069 2023-04-13 14:55:09,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 14:55:09,858 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/9b0c526e1a7c42bc8a7817e140b6d468 2023-04-13 14:55:09,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 14:55:10,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/2f34e311f9874aecb11b5abf1b6e37e4 2023-04-13 14:55:10,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/A/194a7f2eed6e48bd8337a6f4b9b02069 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/194a7f2eed6e48bd8337a6f4b9b02069 2023-04-13 14:55:10,273 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/194a7f2eed6e48bd8337a6f4b9b02069, entries=150, sequenceid=685, filesize=11.9 K 2023-04-13 14:55:10,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/B/9b0c526e1a7c42bc8a7817e140b6d468 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/9b0c526e1a7c42bc8a7817e140b6d468 2023-04-13 14:55:10,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/9b0c526e1a7c42bc8a7817e140b6d468, entries=150, sequenceid=685, filesize=11.9 K 2023-04-13 14:55:10,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/.tmp/C/2f34e311f9874aecb11b5abf1b6e37e4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/2f34e311f9874aecb11b5abf1b6e37e4 2023-04-13 14:55:10,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/2f34e311f9874aecb11b5abf1b6e37e4, entries=150, sequenceid=685, filesize=11.9 K 2023-04-13 14:55:10,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for ccd2cd4c855e3a8c5393dd57d5cc737e in 1237ms, sequenceid=685, compaction requested=false 2023-04-13 14:55:10,305 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d1f0cc8e66954a20ab3a7584f370d796, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/bf1116504a6743d8b60fe9fc5a87bc31, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/769c6646eb76421497b04bde9feaf211, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/a65e9f94b48f48f9a6a77956dd9c6d60, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/e9e190d8dccf42f899f0eb317acf6ead, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/4d0becfccc8f46258e5bc5482b78fed0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/c43891b3849143cda93780fb98fd37c3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/237608c95e0740d49614d0979d62ecbb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/0f65d0feed0f4f7a94beaff1878c50a0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/c264d6b9a9694fe9a125f8df52d029e3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d22c14771ac64ab7a10d8020be661642, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/ca2103d9d837486f949e062b5e1a3c83, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/81fbcbc04bf44564b9579f7f0fb861b0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/58b9f59613fc4e67bd90b6f1ea50190d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/598577a1df2e495c84460c16a3420717, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/270616b5d5ba40e19a8764472bce39a9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/78812c8db64b4e79bad905bed164166f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/39820c88dccc41198c131e2ccbe46112, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/fe2a02ce9f1e4d5980663411ad0e419f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/60b57fdada294bd3a1c6e65707637af4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/7434e990e8134659812197b95da8c37e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/bb99fc423cb74a0788381bb5e1620d90, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/8261670a87524e478e7782fa44ce1bd0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/9761eb771b814f0a8a936eeebdce9ee6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/5de07c243e3d44df92458febe5ceaf52, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d6597d5c65a44dda9863b8e032746c4f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/b01eeca06b8c482c927022c546ca901a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/6f66124b23164344b3a1c601fd7b3cef, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/1b9db0f013d440f48289e2ddafc49f8d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/fe1917e918c64e448b20eee744ec5011, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/180b21d03dca4c1a9c58ea15e09ca1a7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/906587c272a34135af265c07d51ab6da, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/62d86574201447cfadb46a9cb1ca0c89, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/11bdec9adab54cc6891f2b5905fc243e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/6eba45465b3a44609dac4bdcd87da2e7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/30f5efe41761461684a742faf496127c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/540c644dbcce40748187d1e1aa86bf28] to archive 2023-04-13 14:55:10,306 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:55:10,307 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d1f0cc8e66954a20ab3a7584f370d796 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d1f0cc8e66954a20ab3a7584f370d796 2023-04-13 14:55:10,308 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/bf1116504a6743d8b60fe9fc5a87bc31 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/bf1116504a6743d8b60fe9fc5a87bc31 2023-04-13 14:55:10,310 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/769c6646eb76421497b04bde9feaf211 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/769c6646eb76421497b04bde9feaf211 2023-04-13 14:55:10,311 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/a65e9f94b48f48f9a6a77956dd9c6d60 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/a65e9f94b48f48f9a6a77956dd9c6d60 2023-04-13 14:55:10,312 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/e9e190d8dccf42f899f0eb317acf6ead to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/e9e190d8dccf42f899f0eb317acf6ead 2023-04-13 14:55:10,313 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/4d0becfccc8f46258e5bc5482b78fed0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/4d0becfccc8f46258e5bc5482b78fed0 2023-04-13 14:55:10,314 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/c43891b3849143cda93780fb98fd37c3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/c43891b3849143cda93780fb98fd37c3 2023-04-13 14:55:10,315 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/237608c95e0740d49614d0979d62ecbb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/237608c95e0740d49614d0979d62ecbb 2023-04-13 14:55:10,316 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/0f65d0feed0f4f7a94beaff1878c50a0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/0f65d0feed0f4f7a94beaff1878c50a0 2023-04-13 14:55:10,317 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/c264d6b9a9694fe9a125f8df52d029e3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/c264d6b9a9694fe9a125f8df52d029e3 2023-04-13 14:55:10,318 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d22c14771ac64ab7a10d8020be661642 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d22c14771ac64ab7a10d8020be661642 2023-04-13 14:55:10,319 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/ca2103d9d837486f949e062b5e1a3c83 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/ca2103d9d837486f949e062b5e1a3c83 2023-04-13 14:55:10,320 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/81fbcbc04bf44564b9579f7f0fb861b0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/81fbcbc04bf44564b9579f7f0fb861b0 2023-04-13 14:55:10,321 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/58b9f59613fc4e67bd90b6f1ea50190d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/58b9f59613fc4e67bd90b6f1ea50190d 2023-04-13 14:55:10,321 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/598577a1df2e495c84460c16a3420717 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/598577a1df2e495c84460c16a3420717 2023-04-13 14:55:10,322 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/270616b5d5ba40e19a8764472bce39a9 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/270616b5d5ba40e19a8764472bce39a9 2023-04-13 14:55:10,323 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/78812c8db64b4e79bad905bed164166f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/78812c8db64b4e79bad905bed164166f 2023-04-13 14:55:10,324 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/39820c88dccc41198c131e2ccbe46112 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/39820c88dccc41198c131e2ccbe46112 2023-04-13 14:55:10,325 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/fe2a02ce9f1e4d5980663411ad0e419f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/fe2a02ce9f1e4d5980663411ad0e419f 2023-04-13 14:55:10,326 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/60b57fdada294bd3a1c6e65707637af4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/60b57fdada294bd3a1c6e65707637af4 2023-04-13 14:55:10,326 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/7434e990e8134659812197b95da8c37e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/7434e990e8134659812197b95da8c37e 2023-04-13 14:55:10,327 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/bb99fc423cb74a0788381bb5e1620d90 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/bb99fc423cb74a0788381bb5e1620d90 2023-04-13 14:55:10,328 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/8261670a87524e478e7782fa44ce1bd0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/8261670a87524e478e7782fa44ce1bd0 2023-04-13 14:55:10,329 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/9761eb771b814f0a8a936eeebdce9ee6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/9761eb771b814f0a8a936eeebdce9ee6 2023-04-13 14:55:10,330 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/5de07c243e3d44df92458febe5ceaf52 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/5de07c243e3d44df92458febe5ceaf52 2023-04-13 14:55:10,331 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d6597d5c65a44dda9863b8e032746c4f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/d6597d5c65a44dda9863b8e032746c4f 2023-04-13 14:55:10,332 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/b01eeca06b8c482c927022c546ca901a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/b01eeca06b8c482c927022c546ca901a 2023-04-13 14:55:10,332 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/6f66124b23164344b3a1c601fd7b3cef to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/6f66124b23164344b3a1c601fd7b3cef 2023-04-13 14:55:10,333 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/1b9db0f013d440f48289e2ddafc49f8d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/1b9db0f013d440f48289e2ddafc49f8d 2023-04-13 14:55:10,334 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/fe1917e918c64e448b20eee744ec5011 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/fe1917e918c64e448b20eee744ec5011 2023-04-13 14:55:10,335 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/180b21d03dca4c1a9c58ea15e09ca1a7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/180b21d03dca4c1a9c58ea15e09ca1a7 2023-04-13 14:55:10,336 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/906587c272a34135af265c07d51ab6da to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/906587c272a34135af265c07d51ab6da 2023-04-13 14:55:10,337 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/62d86574201447cfadb46a9cb1ca0c89 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/62d86574201447cfadb46a9cb1ca0c89 2023-04-13 14:55:10,338 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/11bdec9adab54cc6891f2b5905fc243e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/11bdec9adab54cc6891f2b5905fc243e 2023-04-13 14:55:10,339 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/6eba45465b3a44609dac4bdcd87da2e7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/6eba45465b3a44609dac4bdcd87da2e7 2023-04-13 14:55:10,339 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/30f5efe41761461684a742faf496127c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/30f5efe41761461684a742faf496127c 2023-04-13 14:55:10,340 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/540c644dbcce40748187d1e1aa86bf28 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/540c644dbcce40748187d1e1aa86bf28 2023-04-13 14:55:10,343 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/eff61f487c734a628c713b5f82b3ef4e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/94d69f68cf7843948ea1cf87c671f744, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5538dbb055d544f39c7cbfa8212cb68b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e6349e183f0f447cbe1b4d0a33c69b85, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/030a5dce741f4ec08ac8e3a3d844b29a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e94f572aeace4497a7cee829339e8460, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/6cfece40b08042c49bfdb5aef92dcc27, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e497e49a2dd0482d85b6a8b768361d23, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/c2a208edfb394f388c63ff5f6aaa58a6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/805178f4cbcc480da7ed467314029570, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/201f01cbc2534bbdb3e3f34c1dac2681, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/791ffc946ee54a4c879fda2340fdfdc3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/78999fa6d38d4c9a99cae62da7f03519, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/04969316840145adbd733b32504b8841, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/835af86082e84722bbeae714bbba51fa, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/1c4a3cd022e54866b8a12b5bf9e2521c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/70ceef37130943959f8f83453aeeac8e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/078542d46369411e81a0cbc469267178, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/7642f44c4e1342d4a9e0816f64100dd0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5d1055f94c4a41d0beb0a0c9e76e2fe5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/2674bb3343b2454d9178bcc7e1519eb3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/d99bfcffcac9424bb2c80fd7d0ce3457, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f35b375a12f0490fae713e34394e7bf3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0c88a3acb15046ffba3c1f2225c5af4c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/58aecf132b8848ad952b28a74f9b87e6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/129a6568a90a4775b806b23975a4bb59, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f2ed1ebd1b86494b8bb2a056d201ce0c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f771fbff733843188fa1f66d94bfe5c7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/73983eb2cf304968b6b0d9992a5d3b37, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5ec032ddbb0a4ac2852c56769a63efed, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/06e33d2680f341f595fa803ef38446bf, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/ffb009d2650a460aab342da4916b77bf, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0e492ed27d5d492aa31af3fd1a0d4d65, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/392021fea6e74c948f45ce58d2e359c0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0495a2242525416796b510e3267b8b66, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/ee33825ae3564862b8d92071a2ef38d6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/45032a255fb94beaa5859208b3bf3b2e] to archive 2023-04-13 14:55:10,344 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:55:10,345 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/eff61f487c734a628c713b5f82b3ef4e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/eff61f487c734a628c713b5f82b3ef4e 2023-04-13 14:55:10,346 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/94d69f68cf7843948ea1cf87c671f744 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/94d69f68cf7843948ea1cf87c671f744 2023-04-13 14:55:10,347 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5538dbb055d544f39c7cbfa8212cb68b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5538dbb055d544f39c7cbfa8212cb68b 2023-04-13 14:55:10,347 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e6349e183f0f447cbe1b4d0a33c69b85 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e6349e183f0f447cbe1b4d0a33c69b85 2023-04-13 14:55:10,348 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/030a5dce741f4ec08ac8e3a3d844b29a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/030a5dce741f4ec08ac8e3a3d844b29a 2023-04-13 14:55:10,349 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e94f572aeace4497a7cee829339e8460 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e94f572aeace4497a7cee829339e8460 2023-04-13 14:55:10,350 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/6cfece40b08042c49bfdb5aef92dcc27 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/6cfece40b08042c49bfdb5aef92dcc27 2023-04-13 14:55:10,351 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e497e49a2dd0482d85b6a8b768361d23 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/e497e49a2dd0482d85b6a8b768361d23 2023-04-13 14:55:10,352 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/c2a208edfb394f388c63ff5f6aaa58a6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/c2a208edfb394f388c63ff5f6aaa58a6 2023-04-13 14:55:10,353 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/805178f4cbcc480da7ed467314029570 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/805178f4cbcc480da7ed467314029570 2023-04-13 14:55:10,354 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/201f01cbc2534bbdb3e3f34c1dac2681 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/201f01cbc2534bbdb3e3f34c1dac2681 2023-04-13 14:55:10,355 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/791ffc946ee54a4c879fda2340fdfdc3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/791ffc946ee54a4c879fda2340fdfdc3 2023-04-13 14:55:10,356 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/78999fa6d38d4c9a99cae62da7f03519 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/78999fa6d38d4c9a99cae62da7f03519 2023-04-13 14:55:10,357 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/04969316840145adbd733b32504b8841 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/04969316840145adbd733b32504b8841 2023-04-13 14:55:10,358 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/835af86082e84722bbeae714bbba51fa to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/835af86082e84722bbeae714bbba51fa 2023-04-13 14:55:10,358 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/1c4a3cd022e54866b8a12b5bf9e2521c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/1c4a3cd022e54866b8a12b5bf9e2521c 2023-04-13 14:55:10,359 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/70ceef37130943959f8f83453aeeac8e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/70ceef37130943959f8f83453aeeac8e 2023-04-13 14:55:10,360 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/078542d46369411e81a0cbc469267178 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/078542d46369411e81a0cbc469267178 2023-04-13 14:55:10,361 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/7642f44c4e1342d4a9e0816f64100dd0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/7642f44c4e1342d4a9e0816f64100dd0 2023-04-13 14:55:10,362 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5d1055f94c4a41d0beb0a0c9e76e2fe5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5d1055f94c4a41d0beb0a0c9e76e2fe5 2023-04-13 14:55:10,363 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/2674bb3343b2454d9178bcc7e1519eb3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/2674bb3343b2454d9178bcc7e1519eb3 2023-04-13 14:55:10,363 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/d99bfcffcac9424bb2c80fd7d0ce3457 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/d99bfcffcac9424bb2c80fd7d0ce3457 2023-04-13 14:55:10,364 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f35b375a12f0490fae713e34394e7bf3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f35b375a12f0490fae713e34394e7bf3 2023-04-13 14:55:10,365 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0c88a3acb15046ffba3c1f2225c5af4c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0c88a3acb15046ffba3c1f2225c5af4c 2023-04-13 14:55:10,366 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/58aecf132b8848ad952b28a74f9b87e6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/58aecf132b8848ad952b28a74f9b87e6 2023-04-13 14:55:10,367 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/129a6568a90a4775b806b23975a4bb59 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/129a6568a90a4775b806b23975a4bb59 2023-04-13 14:55:10,368 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f2ed1ebd1b86494b8bb2a056d201ce0c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f2ed1ebd1b86494b8bb2a056d201ce0c 2023-04-13 14:55:10,369 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f771fbff733843188fa1f66d94bfe5c7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/f771fbff733843188fa1f66d94bfe5c7 2023-04-13 14:55:10,369 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/73983eb2cf304968b6b0d9992a5d3b37 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/73983eb2cf304968b6b0d9992a5d3b37 2023-04-13 14:55:10,370 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5ec032ddbb0a4ac2852c56769a63efed to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/5ec032ddbb0a4ac2852c56769a63efed 2023-04-13 14:55:10,371 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/06e33d2680f341f595fa803ef38446bf to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/06e33d2680f341f595fa803ef38446bf 2023-04-13 14:55:10,372 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/ffb009d2650a460aab342da4916b77bf to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/ffb009d2650a460aab342da4916b77bf 2023-04-13 14:55:10,373 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0e492ed27d5d492aa31af3fd1a0d4d65 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0e492ed27d5d492aa31af3fd1a0d4d65 2023-04-13 14:55:10,374 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/392021fea6e74c948f45ce58d2e359c0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/392021fea6e74c948f45ce58d2e359c0 2023-04-13 14:55:10,375 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0495a2242525416796b510e3267b8b66 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/0495a2242525416796b510e3267b8b66 2023-04-13 14:55:10,376 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/ee33825ae3564862b8d92071a2ef38d6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/ee33825ae3564862b8d92071a2ef38d6 2023-04-13 14:55:10,376 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/45032a255fb94beaa5859208b3bf3b2e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/45032a255fb94beaa5859208b3bf3b2e 2023-04-13 14:55:10,379 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/86f19b1950b5419e90280b4a43013e62, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/a39d90b3c7de4bc3ab9b878cb8b04520, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/244f58b4b2b54dedab97704f2427d376, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7fdb2dfa1213467fae6eb47f31ed3eb1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/180d16f5e6734e698b57908848a3ab9f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/78c04793bd6447638daee15cba692d96, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/3a205fc1bebc417ca344d88fad6c3a16, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/6b0a16bae25c4ebf901ea1aabff163e7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/c7ef38638c4040308120e967e3d994c2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/c3dc619531dc4f8294877d45c4a9c274, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/57dc49729faa450bb8d37e9b31cc81bd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/798053fae5a84e88ab0904b32057b12f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/d1f3c505ee2f4ed0addbad3ac6e6e322, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/f94a511f13724a54af13b996a87ca11a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/42040992c6814e23bd6b19cd534f23f9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/97ca697fc4014f5f98cd2a2a58ec0f5f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/bf33403a916044e9a2f3086ac6b3ebb1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/0ab8f598fdf141ef9927d5615978fddb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7c4872f6cde04cf684f019d1bdc49e2e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/f48b99e3ca6b49999ef7c6cf8eaada67, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7bf83d13637945b5aa4da2eb49076a75, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/b1371e749f914c858db51b2bce0abae3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/24f1ac290c0b4a02a15e03ec0e0a2b3e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/4e066b3ddd4344b7a282ec4493c28079, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/dec2dd11cc3646f78d2fbf60c2ad3762, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e3aade70a8f140f997545efcc1e0d7b0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/bc3b444b5e8f4e878f4f76b47ea9914b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/433aad42ef67468e8b738b3f53051a10, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/19ba66e52e314ffd8d71d5c87560dfbb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/691e2e5e789f467f8b36db1ecd1d66dd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/9d93ce38cc354150979a384f1980f45b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e5ce6a3dc7024b69a84d7be65fb9af2e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/642f1f208960451ebe9342141b7cd980, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/6b61d592f58f4488a2fd6141ccfa5ea5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/fdeef05216aa4fcd8591eaa5b32a8114, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/240c5ab302334d5184b4a64d70da8820, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e4dce7d86ab6466799adf9b7b599ec40] to archive 2023-04-13 14:55:10,380 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:55:10,381 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/86f19b1950b5419e90280b4a43013e62 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/86f19b1950b5419e90280b4a43013e62 2023-04-13 14:55:10,382 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/a39d90b3c7de4bc3ab9b878cb8b04520 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/a39d90b3c7de4bc3ab9b878cb8b04520 2023-04-13 14:55:10,383 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/244f58b4b2b54dedab97704f2427d376 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/244f58b4b2b54dedab97704f2427d376 2023-04-13 14:55:10,384 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7fdb2dfa1213467fae6eb47f31ed3eb1 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7fdb2dfa1213467fae6eb47f31ed3eb1 2023-04-13 14:55:10,385 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/180d16f5e6734e698b57908848a3ab9f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/180d16f5e6734e698b57908848a3ab9f 2023-04-13 14:55:10,386 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/78c04793bd6447638daee15cba692d96 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/78c04793bd6447638daee15cba692d96 2023-04-13 14:55:10,386 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/3a205fc1bebc417ca344d88fad6c3a16 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/3a205fc1bebc417ca344d88fad6c3a16 2023-04-13 14:55:10,387 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/6b0a16bae25c4ebf901ea1aabff163e7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/6b0a16bae25c4ebf901ea1aabff163e7 2023-04-13 14:55:10,388 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/c7ef38638c4040308120e967e3d994c2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/c7ef38638c4040308120e967e3d994c2 2023-04-13 14:55:10,389 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/c3dc619531dc4f8294877d45c4a9c274 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/c3dc619531dc4f8294877d45c4a9c274 2023-04-13 14:55:10,390 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/57dc49729faa450bb8d37e9b31cc81bd to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/57dc49729faa450bb8d37e9b31cc81bd 2023-04-13 14:55:10,390 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/798053fae5a84e88ab0904b32057b12f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/798053fae5a84e88ab0904b32057b12f 2023-04-13 14:55:10,391 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/d1f3c505ee2f4ed0addbad3ac6e6e322 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/d1f3c505ee2f4ed0addbad3ac6e6e322 2023-04-13 14:55:10,392 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/f94a511f13724a54af13b996a87ca11a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/f94a511f13724a54af13b996a87ca11a 2023-04-13 14:55:10,393 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/42040992c6814e23bd6b19cd534f23f9 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/42040992c6814e23bd6b19cd534f23f9 2023-04-13 14:55:10,394 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/97ca697fc4014f5f98cd2a2a58ec0f5f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/97ca697fc4014f5f98cd2a2a58ec0f5f 2023-04-13 14:55:10,395 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/bf33403a916044e9a2f3086ac6b3ebb1 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/bf33403a916044e9a2f3086ac6b3ebb1 2023-04-13 14:55:10,396 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/0ab8f598fdf141ef9927d5615978fddb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/0ab8f598fdf141ef9927d5615978fddb 2023-04-13 14:55:10,397 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7c4872f6cde04cf684f019d1bdc49e2e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7c4872f6cde04cf684f019d1bdc49e2e 2023-04-13 14:55:10,398 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/f48b99e3ca6b49999ef7c6cf8eaada67 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/f48b99e3ca6b49999ef7c6cf8eaada67 2023-04-13 14:55:10,399 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7bf83d13637945b5aa4da2eb49076a75 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/7bf83d13637945b5aa4da2eb49076a75 2023-04-13 14:55:10,400 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/b1371e749f914c858db51b2bce0abae3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/b1371e749f914c858db51b2bce0abae3 2023-04-13 14:55:10,401 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/24f1ac290c0b4a02a15e03ec0e0a2b3e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/24f1ac290c0b4a02a15e03ec0e0a2b3e 2023-04-13 14:55:10,402 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/4e066b3ddd4344b7a282ec4493c28079 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/4e066b3ddd4344b7a282ec4493c28079 2023-04-13 14:55:10,403 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/dec2dd11cc3646f78d2fbf60c2ad3762 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/dec2dd11cc3646f78d2fbf60c2ad3762 2023-04-13 14:55:10,404 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e3aade70a8f140f997545efcc1e0d7b0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e3aade70a8f140f997545efcc1e0d7b0 2023-04-13 14:55:10,405 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/bc3b444b5e8f4e878f4f76b47ea9914b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/bc3b444b5e8f4e878f4f76b47ea9914b 2023-04-13 14:55:10,406 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/433aad42ef67468e8b738b3f53051a10 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/433aad42ef67468e8b738b3f53051a10 2023-04-13 14:55:10,407 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/19ba66e52e314ffd8d71d5c87560dfbb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/19ba66e52e314ffd8d71d5c87560dfbb 2023-04-13 14:55:10,408 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/691e2e5e789f467f8b36db1ecd1d66dd to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/691e2e5e789f467f8b36db1ecd1d66dd 2023-04-13 14:55:10,410 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/9d93ce38cc354150979a384f1980f45b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/9d93ce38cc354150979a384f1980f45b 2023-04-13 14:55:10,411 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e5ce6a3dc7024b69a84d7be65fb9af2e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e5ce6a3dc7024b69a84d7be65fb9af2e 2023-04-13 14:55:10,412 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/642f1f208960451ebe9342141b7cd980 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/642f1f208960451ebe9342141b7cd980 2023-04-13 14:55:10,414 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/6b61d592f58f4488a2fd6141ccfa5ea5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/6b61d592f58f4488a2fd6141ccfa5ea5 2023-04-13 14:55:10,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/fdeef05216aa4fcd8591eaa5b32a8114 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/fdeef05216aa4fcd8591eaa5b32a8114 2023-04-13 14:55:10,416 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/240c5ab302334d5184b4a64d70da8820 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/240c5ab302334d5184b4a64d70da8820 2023-04-13 14:55:10,418 DEBUG [StoreCloser-TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e4dce7d86ab6466799adf9b7b599ec40 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/e4dce7d86ab6466799adf9b7b599ec40 2023-04-13 14:55:10,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/recovered.edits/688.seqid, newMaxSeqId=688, maxSeqId=1 2023-04-13 14:55:10,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e. 2023-04-13 14:55:10,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ccd2cd4c855e3a8c5393dd57d5cc737e: 2023-04-13 14:55:10,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:55:10,425 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=ccd2cd4c855e3a8c5393dd57d5cc737e, regionState=CLOSED 2023-04-13 14:55:10,425 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397710425"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397710425"}]},"ts":"1681397710425"} 2023-04-13 14:55:10,427 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-13 14:55:10,427 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure ccd2cd4c855e3a8c5393dd57d5cc737e, server=jenkins-hbase4.apache.org,43875,1681397617638 in 1.5340 sec 2023-04-13 14:55:10,428 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-13 14:55:10,428 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ccd2cd4c855e3a8c5393dd57d5cc737e, UNASSIGN in 1.5380 sec 2023-04-13 14:55:10,429 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397710429"}]},"ts":"1681397710429"} 2023-04-13 14:55:10,429 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 14:55:10,431 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 14:55:10,432 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5490 sec 2023-04-13 14:55:10,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 14:55:10,990 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-13 14:55:10,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 14:55:10,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:10,993 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:10,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-13 14:55:10,993 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:10,997 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:55:10,998 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/recovered.edits] 2023-04-13 14:55:11,001 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/194a7f2eed6e48bd8337a6f4b9b02069 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/194a7f2eed6e48bd8337a6f4b9b02069 2023-04-13 14:55:11,002 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/9218535f10d84a47997dda11a8d4e725 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/A/9218535f10d84a47997dda11a8d4e725 2023-04-13 14:55:11,004 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/9b0c526e1a7c42bc8a7817e140b6d468 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/9b0c526e1a7c42bc8a7817e140b6d468 2023-04-13 14:55:11,005 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/cb9efb3949ea426f82ec1a1d712c2024 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/B/cb9efb3949ea426f82ec1a1d712c2024 2023-04-13 14:55:11,006 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/2f34e311f9874aecb11b5abf1b6e37e4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/2f34e311f9874aecb11b5abf1b6e37e4 2023-04-13 14:55:11,007 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/5db4ccbb020f4b2da619c8ae21476d8b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/C/5db4ccbb020f4b2da619c8ae21476d8b 2023-04-13 14:55:11,009 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/recovered.edits/688.seqid to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e/recovered.edits/688.seqid 2023-04-13 14:55:11,010 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/ccd2cd4c855e3a8c5393dd57d5cc737e 2023-04-13 14:55:11,010 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 14:55:11,012 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:11,013 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 14:55:11,014 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 14:55:11,015 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:11,015 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 14:55:11,015 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681397711015"}]},"ts":"9223372036854775807"} 2023-04-13 14:55:11,016 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 14:55:11,016 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ccd2cd4c855e3a8c5393dd57d5cc737e, NAME => 'TestAcidGuarantees,,1681397681646.ccd2cd4c855e3a8c5393dd57d5cc737e.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 14:55:11,016 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 14:55:11,016 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681397711016"}]},"ts":"9223372036854775807"} 2023-04-13 14:55:11,018 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 14:55:11,019 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:11,020 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 28 msec 2023-04-13 14:55:11,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-13 14:55:11,094 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-13 14:55:11,105 INFO [Listener at localhost/37101] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227 (was 227), OpenFileDescriptor=558 (was 531) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=298 (was 356), ProcessCount=169 (was 169), AvailableMemoryMB=779 (was 814) 2023-04-13 14:55:11,116 INFO [Listener at localhost/37101] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227, OpenFileDescriptor=558, MaxFileDescriptor=60000, SystemLoadAverage=298, ProcessCount=169, AvailableMemoryMB=778 2023-04-13 14:55:11,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 14:55:11,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 14:55:11,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:11,120 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 14:55:11,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-13 14:55:11,120 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 14:55:11,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 14:55:11,121 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:11,122 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1 empty. 2023-04-13 14:55:11,122 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:11,122 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 14:55:11,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 14:55:11,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 14:55:11,529 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 14:55:11,530 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7763c91103ebbe310342746ed8aa8aa1, NAME => 'TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp 2023-04-13 14:55:11,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 14:55:11,938 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:55:11,938 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 7763c91103ebbe310342746ed8aa8aa1, disabling compactions & flushes 2023-04-13 14:55:11,938 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:11,938 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:11,938 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. after waiting 0 ms 2023-04-13 14:55:11,938 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:11,938 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:11,938 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:11,940 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 14:55:11,941 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397711941"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397711941"}]},"ts":"1681397711941"} 2023-04-13 14:55:11,942 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 14:55:11,942 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 14:55:11,942 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397711942"}]},"ts":"1681397711942"} 2023-04-13 14:55:11,943 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 14:55:11,947 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7763c91103ebbe310342746ed8aa8aa1, ASSIGN}] 2023-04-13 14:55:11,949 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7763c91103ebbe310342746ed8aa8aa1, ASSIGN 2023-04-13 14:55:11,949 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7763c91103ebbe310342746ed8aa8aa1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43875,1681397617638; forceNewPlan=false, retain=false 2023-04-13 14:55:12,100 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=7763c91103ebbe310342746ed8aa8aa1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:12,100 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397712100"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397712100"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397712100"}]},"ts":"1681397712100"} 2023-04-13 14:55:12,102 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:55:12,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 14:55:12,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:12,256 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7763c91103ebbe310342746ed8aa8aa1, NAME => 'TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.', STARTKEY => '', ENDKEY => ''} 2023-04-13 14:55:12,256 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:12,256 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:55:12,256 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:12,256 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:12,257 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:12,258 DEBUG [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A 2023-04-13 14:55:12,259 DEBUG [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A 2023-04-13 14:55:12,259 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:55:12,259 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7763c91103ebbe310342746ed8aa8aa1 columnFamilyName A 2023-04-13 14:55:12,259 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.HStore(310): Store=7763c91103ebbe310342746ed8aa8aa1/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:55:12,259 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:12,260 DEBUG [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B 2023-04-13 14:55:12,260 DEBUG [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B 2023-04-13 14:55:12,261 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:55:12,261 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7763c91103ebbe310342746ed8aa8aa1 columnFamilyName B 2023-04-13 14:55:12,261 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.HStore(310): Store=7763c91103ebbe310342746ed8aa8aa1/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:55:12,261 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:12,262 DEBUG [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C 2023-04-13 14:55:12,262 DEBUG [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C 2023-04-13 14:55:12,262 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:55:12,262 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7763c91103ebbe310342746ed8aa8aa1 columnFamilyName C 2023-04-13 14:55:12,263 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.HStore(310): Store=7763c91103ebbe310342746ed8aa8aa1/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:55:12,263 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:12,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:12,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:12,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 14:55:12,266 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:12,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 14:55:12,268 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7763c91103ebbe310342746ed8aa8aa1; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61319188, jitterRate=-0.08627289533615112}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 14:55:12,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:12,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., pid=37, masterSystemTime=1681397712253 2023-04-13 14:55:12,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:12,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:12,270 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=7763c91103ebbe310342746ed8aa8aa1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:12,270 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397712270"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681397712270"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681397712270"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681397712270"}]},"ts":"1681397712270"} 2023-04-13 14:55:12,272 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-13 14:55:12,272 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 in 170 msec 2023-04-13 14:55:12,273 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-13 14:55:12,274 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7763c91103ebbe310342746ed8aa8aa1, ASSIGN in 325 msec 2023-04-13 14:55:12,274 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 14:55:12,274 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397712274"}]},"ts":"1681397712274"} 2023-04-13 14:55:12,275 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 14:55:12,277 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 14:55:12,277 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1590 sec 2023-04-13 14:55:13,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 14:55:13,225 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-13 14:55:13,226 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x4473f45c to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:13,230 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fd3d54e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:13,232 DEBUG [Listener at localhost/37101] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:13,233 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39250, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:13,234 DEBUG [Listener at localhost/37101] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 14:55:13,235 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41084, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 14:55:13,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 14:55:13,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 14:55:13,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:13,649 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 14:55:13,650 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 14:55:13,650 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 14:55:13,652 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-13 14:55:13,654 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7763c91103ebbe310342746ed8aa8aa1, REOPEN/MOVE}] 2023-04-13 14:55:13,655 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7763c91103ebbe310342746ed8aa8aa1, REOPEN/MOVE 2023-04-13 14:55:13,655 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7763c91103ebbe310342746ed8aa8aa1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:13,656 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397713655"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397713655"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397713655"}]},"ts":"1681397713655"} 2023-04-13 14:55:13,657 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:55:13,808 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:13,809 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7763c91103ebbe310342746ed8aa8aa1, disabling compactions & flushes 2023-04-13 14:55:13,809 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:13,809 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:13,809 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. after waiting 0 ms 2023-04-13 14:55:13,809 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:13,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-13 14:55:13,813 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:13,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:13,813 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 7763c91103ebbe310342746ed8aa8aa1 to self. 2023-04-13 14:55:13,814 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:13,814 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7763c91103ebbe310342746ed8aa8aa1, regionState=CLOSED 2023-04-13 14:55:13,815 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397713814"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397713814"}]},"ts":"1681397713814"} 2023-04-13 14:55:13,816 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-13 14:55:13,816 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 in 159 msec 2023-04-13 14:55:13,817 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7763c91103ebbe310342746ed8aa8aa1, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,43875,1681397617638; forceNewPlan=false, retain=true 2023-04-13 14:55:13,967 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7763c91103ebbe310342746ed8aa8aa1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:13,967 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397713967"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397713967"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397713967"}]},"ts":"1681397713967"} 2023-04-13 14:55:13,969 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:55:14,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7763c91103ebbe310342746ed8aa8aa1, NAME => 'TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.', STARTKEY => '', ENDKEY => ''} 2023-04-13 14:55:14,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:14,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:55:14,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:14,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:14,125 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:14,126 DEBUG [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A 2023-04-13 14:55:14,126 DEBUG [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A 2023-04-13 14:55:14,126 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:55:14,126 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7763c91103ebbe310342746ed8aa8aa1 columnFamilyName A 2023-04-13 14:55:14,128 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.HStore(310): Store=7763c91103ebbe310342746ed8aa8aa1/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:55:14,128 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:14,129 DEBUG [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B 2023-04-13 14:55:14,129 DEBUG [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B 2023-04-13 14:55:14,129 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:55:14,129 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7763c91103ebbe310342746ed8aa8aa1 columnFamilyName B 2023-04-13 14:55:14,129 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.HStore(310): Store=7763c91103ebbe310342746ed8aa8aa1/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:55:14,129 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:14,130 DEBUG [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C 2023-04-13 14:55:14,130 DEBUG [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C 2023-04-13 14:55:14,130 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:55:14,130 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7763c91103ebbe310342746ed8aa8aa1 columnFamilyName C 2023-04-13 14:55:14,131 INFO [StoreOpener-7763c91103ebbe310342746ed8aa8aa1-1] regionserver.HStore(310): Store=7763c91103ebbe310342746ed8aa8aa1/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:55:14,131 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,132 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:14,132 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:14,134 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 14:55:14,135 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:14,135 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7763c91103ebbe310342746ed8aa8aa1; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67443419, jitterRate=0.004985257983207703}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 14:55:14,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:14,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., pid=42, masterSystemTime=1681397714120 2023-04-13 14:55:14,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,137 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,138 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7763c91103ebbe310342746ed8aa8aa1, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,138 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397714137"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681397714137"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681397714137"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681397714137"}]},"ts":"1681397714137"} 2023-04-13 14:55:14,139 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-13 14:55:14,139 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 in 169 msec 2023-04-13 14:55:14,140 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-13 14:55:14,141 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7763c91103ebbe310342746ed8aa8aa1, REOPEN/MOVE in 485 msec 2023-04-13 14:55:14,142 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-13 14:55:14,142 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-04-13 14:55:14,143 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 905 msec 2023-04-13 14:55:14,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-13 14:55:14,145 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x41507d16 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:14,149 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a5ab074, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:14,150 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x49b5bd0d to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:14,154 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bcfc1e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:14,155 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x5c8d927d to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:14,158 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38341134, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:14,158 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x419b69bf to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:14,161 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71b5058b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:14,162 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x2c6cac96 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:14,165 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3620ee35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:14,165 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe1db6a to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:14,174 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e01d5e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:14,174 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x10377b50 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:14,178 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ccd8b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:14,178 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x4366fcdd to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:14,183 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68a0e21a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:14,183 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x6a35a28d to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:14,187 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ec12b02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:14,187 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x5a1d4246 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:14,192 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d7ba242, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:14,197 DEBUG [hconnection-0x1bf74800-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:14,197 DEBUG [hconnection-0x1bdd631e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:14,197 DEBUG [hconnection-0x2aece17c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:14,197 DEBUG [hconnection-0x45ae402c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:14,197 DEBUG [hconnection-0xacef1fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:14,198 DEBUG [hconnection-0x493eabad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:14,199 DEBUG [hconnection-0x70a06e9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:14,199 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:14,199 DEBUG [hconnection-0x204110e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:14,199 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57278, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:14,200 DEBUG [hconnection-0x397821aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:14,200 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57332, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:14,205 DEBUG [hconnection-0x734984b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:14,208 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57288, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:14,208 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57290, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:14,209 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:14,209 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57326, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:14,209 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57336, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:14,209 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57350, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:14,209 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:14,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:14,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:14,224 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 14:55:14,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:14,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:14,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:14,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:14,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:14,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:14,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:14,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:14,241 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:14,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:14,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:14,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:14,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,242 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,242 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:14,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:14,242 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,242 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:14,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:14,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:14,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:14,245 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:14,245 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:14,245 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:14,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:14,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:14,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:14,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:14,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:14,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:14,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:14,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:14,249 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,249 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,250 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,250 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:14,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59030799[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:14,251 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:14,251 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,252 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:14,253 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:14,253 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:14,253 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,253 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:14,255 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:14,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:14,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:14,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:14,257 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:14,257 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:14,257 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:14,257 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:14,257 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:14,258 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,258 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,258 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:14,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,259 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,260 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,262 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:14,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:14,262 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:14,262 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,263 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,263 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,263 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,263 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,263 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,264 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:14,264 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:14,270 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:14,270 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:14,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:14,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:14,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:14,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:14,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397774262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397774262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397774264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397774268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397774271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,364 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:14,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:14,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:14,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:14,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:14,370 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:14,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:14,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:14,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:14,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,372 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:14,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:14,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:14,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397774372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,373 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,373 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:14,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397774372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397774372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397774373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397774373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:14,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:14,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:14,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:14,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:14,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:14,376 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:14,380 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,380 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:14,380 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:14,380 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:14,383 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:14,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:14,385 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:14,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46d9d8a8[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:14,385 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,386 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,386 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:14,386 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:14,386 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:14,387 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,387 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:14,388 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,389 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,389 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:14,389 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:14,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:14,392 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:14,392 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:14,392 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:14,392 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:14,392 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:14,393 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,393 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,393 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:14,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,394 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:14,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:14,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,403 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:14,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:14,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:14,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:14,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:14,404 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:14,404 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:14,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:14,504 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:14,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:14,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:14,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:14,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:14,507 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:14,507 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:14,507 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:14,507 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:14,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,514 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:14,514 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:14,515 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,515 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:14,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:14,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:14,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:14,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:14,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:14,520 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:14,520 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:14,520 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,520 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:14,520 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:14,520 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,520 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:14,520 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:14,520 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,520 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:14,520 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:14,522 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:14,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e737cc0[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:14,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:14,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,525 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:14,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:14,525 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,525 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:14,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:14,528 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:14,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:14,530 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:14,530 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:14,530 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:14,530 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:14,530 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:14,531 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:14,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:14,532 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,533 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,535 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,535 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:14,535 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,535 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:14,535 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:14,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:14,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:14,535 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:14,535 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,536 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,536 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,536 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,536 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:14,536 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:14,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:14,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:14,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:14,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397774574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397774574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397774574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397774575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397774575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,636 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:14,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:14,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:14,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:14,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:14,639 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:14,639 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:14,640 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:14,640 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:14,641 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,641 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,642 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:14,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:14,642 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,642 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:14,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:14,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:14,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:14,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:14,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:14,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:14,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:14,644 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,644 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:14,644 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:14,644 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:14,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:14,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:14,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:14,646 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,646 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,647 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,647 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:14,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24af018d[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:14,649 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:14,649 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,650 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,651 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:14,651 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:14,651 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:14,651 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,651 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:14,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:14,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:14,652 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:14,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:14,654 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:14,655 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:14,655 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:14,655 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:14,655 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:14,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:14,657 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,658 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:14,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:14,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:14,661 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:14,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:14,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:14,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:14,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:14,661 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:14,661 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:14,667 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413265a7615bef3419f8ece10c1db69dc16 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413265a7615bef3419f8ece10c1db69dc16 2023-04-13 14:55:14,761 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:14,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:14,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:14,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:14,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:14,765 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:14,765 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:14,765 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:14,765 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:14,765 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,765 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,767 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:14,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:14,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:14,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:14,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:14,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:14,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:14,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:14,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:14,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:14,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:14,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:14,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:14,768 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,768 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:14,769 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:14,769 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:14,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:14,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:14,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:14,770 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,770 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,771 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,771 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:14,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6551e54e[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:14,772 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:14,772 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,773 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,774 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:14,774 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:14,774 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:14,774 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,774 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:14,776 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,777 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:14,777 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:14,777 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:14,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:14,778 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:14,779 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:14,779 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:14,779 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:14,779 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:14,780 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,780 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,780 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,781 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,782 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,785 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:14,785 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:14,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:14,785 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,785 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,785 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:14,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:14,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:14,785 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,785 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:14,785 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:14,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:14,785 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:14,785 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,786 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,786 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,886 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:14,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:14,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397774882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397774882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397774883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:14,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397774883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:14,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397774883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:14,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:14,889 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:14,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:14,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:14,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:14,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,891 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,891 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:14,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:14,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:14,891 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,891 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:14,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:14,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:14,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:14,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:14,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:14,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:14,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:14,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:14,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:14,892 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,892 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:14,892 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:14,892 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:14,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:14,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:14,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:14,893 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:14,897 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,897 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,897 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,897 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:14,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dc56dd4[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:14,899 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:14,899 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,902 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:14,903 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:14,903 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:14,903 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,903 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:14,904 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:14,904 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,904 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,904 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:14,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:14,906 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:14,906 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:14,906 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:14,906 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:14,906 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:14,909 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,909 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:14,909 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:14,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,911 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,912 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,914 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:14,914 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,915 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:14,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:14,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:14,914 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:14,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:14,915 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:14,915 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:14,915 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:14,915 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:14,915 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:14,915 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,915 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:14,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:14,915 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,015 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:15,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:15,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:15,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:15,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:15,019 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:15,019 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:15,019 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:15,019 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:15,019 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,019 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,022 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,022 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:15,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:15,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:15,022 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,022 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:15,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:15,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:15,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:15,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:15,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:15,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:15,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:15,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:15,023 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:15,024 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:15,024 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:15,024 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:15,025 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,025 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:15,026 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,026 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:15,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:15,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18a599d0[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:15,027 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:15,027 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,028 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,028 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:15,028 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:15,028 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:15,028 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,029 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:15,030 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,031 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:15,031 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:15,031 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:15,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:15,032 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:15,032 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:15,032 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:15,032 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:15,032 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:15,035 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,035 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:15,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:15,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,035 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:15,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:15,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:15,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:15,035 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,038 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:15,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,040 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,040 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:15,040 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:15,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:15,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:15,040 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,040 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:15,040 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:15,041 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:15,041 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:15,040 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:15,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:15,041 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,041 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,041 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:15,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:15,072 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/4c35de3c604946a59adcaaff80dc2f21 2023-04-13 14:55:15,141 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:15,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:15,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:15,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:15,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:15,144 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:15,144 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:15,144 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:15,144 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:15,144 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,145 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,146 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:15,146 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:15,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:15,146 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,146 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:15,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:15,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:15,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:15,147 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:15,147 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:15,147 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:15,147 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:15,147 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:15,148 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:15,148 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:15,148 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:15,148 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:15,149 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,149 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:15,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:15,150 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,150 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:15,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:15,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44da5020[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:15,151 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:15,151 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,153 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,154 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:15,154 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:15,154 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,154 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:15,154 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:15,155 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,155 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:15,155 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:15,155 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:15,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:15,157 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:15,158 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:15,158 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:15,158 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:15,158 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:15,159 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,159 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:15,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,159 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:15,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,160 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:15,161 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,163 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,163 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:15,163 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:15,163 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:15,163 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:15,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:15,163 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:15,164 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,164 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,164 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:15,164 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:15,164 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:15,164 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,264 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:15,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:15,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:15,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:15,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:15,268 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:15,268 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:15,268 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:15,268 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:15,268 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,268 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,270 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,270 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:15,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:15,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:15,271 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,271 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:15,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:15,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:15,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:15,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:15,272 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:15,272 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:15,272 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:15,272 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:15,274 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:15,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:15,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:15,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e4ba660[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:15,276 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:15,276 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,277 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,277 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:15,277 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:15,277 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:15,278 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,278 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:15,282 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,282 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:15,282 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:15,282 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:15,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:15,284 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:15,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:15,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:15,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:15,284 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:15,286 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,286 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:15,286 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:15,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:15,287 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,288 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:15,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,290 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,290 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,290 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,290 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,290 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,290 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,291 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:15,291 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:15,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:15,291 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:15,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:15,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:15,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:15,291 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:15,292 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:15,292 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:15,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:15,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397775388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,392 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:15,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:15,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:15,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397775390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:15,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397775390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:15,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:15,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397775392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:15,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397775393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:15,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:15,396 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:15,396 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:15,396 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:15,396 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:15,397 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,397 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,398 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,398 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:15,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:15,398 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,398 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:15,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:15,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:15,399 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:15,399 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:15,400 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:15,400 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:15,400 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:15,400 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:15,407 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,407 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:15,407 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,407 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:15,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:15,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:15,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6826c5b0[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:15,410 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:15,410 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,411 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:15,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:15,412 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,412 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:15,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:15,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:15,413 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:15,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:15,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:15,416 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:15,416 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:15,416 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:15,416 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:15,416 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:15,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:15,418 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:15,419 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:15,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:15,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:15,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:15,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:15,425 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:15,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:15,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:15,425 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:15,425 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:15,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:15,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:15,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:15,483 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/ac238953e3924f4cbb9c8054bace2140 2023-04-13 14:55:15,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/7a20536cb775409ca10915edefa7e410 2023-04-13 14:55:15,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/4c35de3c604946a59adcaaff80dc2f21 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/4c35de3c604946a59adcaaff80dc2f21 2023-04-13 14:55:15,504 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/4c35de3c604946a59adcaaff80dc2f21, entries=150, sequenceid=17, filesize=25.1 K 2023-04-13 14:55:15,505 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/ac238953e3924f4cbb9c8054bace2140 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/ac238953e3924f4cbb9c8054bace2140 2023-04-13 14:55:15,508 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/ac238953e3924f4cbb9c8054bace2140, entries=150, sequenceid=17, filesize=11.6 K 2023-04-13 14:55:15,509 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/7a20536cb775409ca10915edefa7e410 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/7a20536cb775409ca10915edefa7e410 2023-04-13 14:55:15,513 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/7a20536cb775409ca10915edefa7e410, entries=150, sequenceid=17, filesize=11.6 K 2023-04-13 14:55:15,514 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7763c91103ebbe310342746ed8aa8aa1 in 1290ms, sequenceid=17, compaction requested=false 2023-04-13 14:55:15,514 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:15,526 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:15,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:15,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:15,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:15,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:15,529 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:15,529 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:15,529 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:15,529 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:15,529 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,529 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,530 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:15,530 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:15,530 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:15,530 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:15,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:15,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:15,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:15,531 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:15,531 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:15,531 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:15,532 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:15,532 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:15,532 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:15,532 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:15,532 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 14:55:15,532 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:15,532 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:15,532 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:15,532 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:15,532 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:15,532 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:15,548 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413f64cabdb25bd4471bf115d3d9bd16aef to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f64cabdb25bd4471bf115d3d9bd16aef 2023-04-13 14:55:15,953 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/8f9b834f14f84699b4e838f500842749 2023-04-13 14:55:15,954 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 14:55:15,992 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/51e8cb4e0e194422896216d5ded96ecb 2023-04-13 14:55:16,004 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/916a970a04444735996ba588d4cbbac1 2023-04-13 14:55:16,009 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/8f9b834f14f84699b4e838f500842749 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/8f9b834f14f84699b4e838f500842749 2023-04-13 14:55:16,014 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/8f9b834f14f84699b4e838f500842749, entries=150, sequenceid=36, filesize=25.1 K 2023-04-13 14:55:16,015 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/51e8cb4e0e194422896216d5ded96ecb as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/51e8cb4e0e194422896216d5ded96ecb 2023-04-13 14:55:16,020 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/51e8cb4e0e194422896216d5ded96ecb, entries=150, sequenceid=36, filesize=11.6 K 2023-04-13 14:55:16,021 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/916a970a04444735996ba588d4cbbac1 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/916a970a04444735996ba588d4cbbac1 2023-04-13 14:55:16,024 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/916a970a04444735996ba588d4cbbac1, entries=150, sequenceid=36, filesize=11.6 K 2023-04-13 14:55:16,025 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 7763c91103ebbe310342746ed8aa8aa1 in 493ms, sequenceid=36, compaction requested=false 2023-04-13 14:55:16,025 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:16,025 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,025 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:16,025 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:16,025 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,025 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:16,025 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,027 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:16,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:16,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c79aa7d[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:16,041 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:16,041 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,042 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,042 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:16,042 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:16,042 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:16,042 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,043 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:16,045 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:16,045 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,045 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,045 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:16,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:16,048 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:16,048 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:16,048 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:16,048 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:16,048 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:16,049 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,049 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,049 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,050 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,051 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,054 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,054 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,054 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,054 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,054 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:16,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:16,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:16,054 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:16,054 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,054 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,054 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:16,054 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:16,054 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:16,054 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,155 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:16,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:16,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:16,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:16,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:16,158 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:16,158 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:16,158 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:16,158 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:16,159 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,159 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,161 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:16,161 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:16,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:16,161 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,161 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:16,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:16,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:16,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:16,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:16,163 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:16,163 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:16,163 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,163 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:16,163 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:16,163 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,163 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:16,163 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:16,163 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,163 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:16,163 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:16,166 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,166 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,166 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,166 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:16,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a34ba82[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:16,168 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:16,168 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,169 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,170 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:16,170 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:16,170 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:16,170 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,170 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:16,171 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:16,171 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,171 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,171 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:16,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:16,175 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:16,176 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:16,176 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:16,176 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:16,176 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:16,177 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,177 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,177 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,179 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,180 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,186 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:16,186 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:16,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:16,186 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,186 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,186 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:16,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:16,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,186 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,186 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:16,186 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,186 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:16,186 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,186 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,186 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,286 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:16,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:16,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:16,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:16,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:16,290 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:16,290 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:16,290 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:16,290 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:16,291 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,291 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,292 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,292 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:16,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:16,292 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,292 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:16,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:16,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:16,294 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,294 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:16,294 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:16,294 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:16,296 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,296 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,296 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,296 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:16,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@393f7d02[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:16,298 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:16,298 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,300 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:16,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:16,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:16,300 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,300 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:16,302 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,302 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:16,302 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,302 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:16,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:16,310 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:16,310 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:16,310 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:16,310 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:16,310 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:16,312 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,312 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,314 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,315 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,317 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,319 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,319 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,319 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,319 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:16,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:16,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:16,319 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:16,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:16,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:16,319 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,319 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,320 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,320 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:16,320 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,320 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:16,320 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,420 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:16,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:16,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:16,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:16,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:16,424 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:16,424 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:16,424 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:16,424 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:16,425 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,425 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,427 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,427 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:16,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:16,427 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,427 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:16,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:16,429 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:16,429 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:16,429 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:16,432 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:16,432 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:16,432 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,432 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:16,432 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:16,432 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,433 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:16,433 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:16,433 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,433 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:16,433 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:16,434 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,434 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,434 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,434 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:16,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67ea24d0[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:16,436 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:16,436 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,437 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,437 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:16,437 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:16,437 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:16,437 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,437 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:16,438 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:16,439 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,439 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:16,439 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:16,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:16,441 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:16,442 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:16,442 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:16,442 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:16,442 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:16,444 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,444 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,444 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,445 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,446 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,448 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:16,448 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:16,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:16,448 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,449 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,449 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:16,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:16,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:16,449 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:16,449 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:16,449 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,449 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,449 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,449 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,449 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:16,491 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:55:16,492 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:16,492 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:16,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:16,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:16,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:16,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:16,519 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413d4c10ff3ecba49ad8cf90125142f748d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d4c10ff3ecba49ad8cf90125142f748d 2023-04-13 14:55:16,527 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/91166ef7c7d74113a74feced3dcfa4de 2023-04-13 14:55:16,549 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:16,550 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/7743183e0d0b43889df07a512fc3e8c4 2023-04-13 14:55:16,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:16,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681397716493, max timestamp=1681397716549Num uniques 859; 2023-04-13 14:55:16,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-13 14:55:16,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681397716493, max timestamp=1681397716549Num uniques 859; 2023-04-13 14:55:16,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-13 14:55:16,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681397716493, max timestamp=1681397716549Num uniques 859; 2023-04-13 14:55:16,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-13 14:55:16,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:16,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:16,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:16,555 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:16,555 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:16,555 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:16,555 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:16,560 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,560 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:16,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397776552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:16,561 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:16,561 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:16,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397776554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,562 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,562 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:16,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397776556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:16,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:16,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:16,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397776557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:16,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:16,568 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:16,568 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:16,568 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:16,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:16,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397776560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,571 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:16,571 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:16,571 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,571 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:16,571 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:16,572 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:16,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,573 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,574 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,574 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:16,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76d14012[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:16,575 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:16,575 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,580 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,580 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:16,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:16,580 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:16,581 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:16,582 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,582 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:16,582 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,582 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:16,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:16,585 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:16,585 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:16,585 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:16,585 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:16,585 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:16,586 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,586 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,586 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,587 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,588 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,593 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,593 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:16,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:16,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:16,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:16,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:16,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,594 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,596 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:16,596 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:16,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:16,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397776663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:16,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397776663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:16,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397776663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:16,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397776668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:16,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397776669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,696 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:16,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:16,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:16,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:16,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:16,699 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:16,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:16,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:16,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:16,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,700 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:16,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:16,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:16,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,702 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,702 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:16,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:16,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:16,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:16,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:16,702 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,703 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:16,703 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:16,703 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,703 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:16,703 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:16,703 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,703 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:16,703 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:16,704 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:16,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@195e04e0[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:16,706 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:16,706 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,708 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,708 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:16,708 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:16,708 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,708 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:16,708 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:16,709 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:16,709 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:16,711 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:16,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:16,713 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:16,713 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:16,713 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:16,713 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:16,713 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:16,715 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,715 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,715 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,715 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,716 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,718 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,718 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,718 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,718 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:16,718 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:16,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:16,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:16,718 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,718 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:16,719 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,719 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:16,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,719 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,719 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:16,719 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,819 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:16,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:16,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:16,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:16,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:16,822 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:16,822 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:16,822 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:16,822 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:16,823 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,823 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,825 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,825 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:16,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:16,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:16,825 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,825 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:16,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:16,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:16,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:16,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:16,826 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:16,826 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:16,826 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:16,826 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:16,826 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:16,827 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,827 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:16,827 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:16,827 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,827 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:16,827 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:16,827 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,827 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:16,827 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:16,828 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,828 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,829 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,829 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:16,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:16,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bcbbda4[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:16,830 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:16,830 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,831 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:16,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:16,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:16,831 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,831 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:16,832 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,833 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:16,833 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,833 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:16,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:16,835 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:16,835 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:16,835 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:16,835 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:16,835 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:16,837 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,837 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,837 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,838 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,839 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,841 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,841 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,841 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:16,841 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:16,841 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:16,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:16,841 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:16,842 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,842 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,842 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,842 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,842 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:16,842 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:16,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:16,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397776865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:16,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397776865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:16,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397776865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:16,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397776872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:16,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397776872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,942 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:16,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:16,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:16,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:16,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:16,945 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:16,946 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:16,946 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:16,946 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:16,946 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,946 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,947 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:16,947 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:16,948 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,948 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:16,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:16,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:16,949 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,949 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:16,949 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:16,949 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:16,951 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,951 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,951 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,951 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:16,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d31dea9[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:16,953 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:16,953 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,955 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,955 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:16,955 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,955 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:16,955 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:16,955 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:16,956 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,957 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:16,957 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,957 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:16,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:16,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:16,959 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:16,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:16,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:16,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:16,959 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:16,960 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,961 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:16,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:16,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,961 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:16,962 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,963 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:16,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,965 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,965 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,965 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:16,965 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:16,965 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:16,965 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:16,965 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:16,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:16,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:16,965 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:16,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:16,965 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:16,966 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:16,966 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:16,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:16,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:16,966 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:16,973 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/b295aa232cba4c058c5cdcc39a784d34 2023-04-13 14:55:16,977 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/91166ef7c7d74113a74feced3dcfa4de as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/91166ef7c7d74113a74feced3dcfa4de 2023-04-13 14:55:16,981 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/91166ef7c7d74113a74feced3dcfa4de, entries=400, sequenceid=50, filesize=59.0 K 2023-04-13 14:55:16,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/7743183e0d0b43889df07a512fc3e8c4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/7743183e0d0b43889df07a512fc3e8c4 2023-04-13 14:55:16,987 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/7743183e0d0b43889df07a512fc3e8c4, entries=400, sequenceid=50, filesize=23.0 K 2023-04-13 14:55:16,988 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/b295aa232cba4c058c5cdcc39a784d34 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b295aa232cba4c058c5cdcc39a784d34 2023-04-13 14:55:16,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b295aa232cba4c058c5cdcc39a784d34, entries=400, sequenceid=50, filesize=23.0 K 2023-04-13 14:55:16,991 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=147.60 KB/151140 for 7763c91103ebbe310342746ed8aa8aa1 in 500ms, sequenceid=50, compaction requested=true 2023-04-13 14:55:16,991 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:16,991 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:16,991 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:16,991 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:16,991 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:16,992 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:16,992 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 111946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:16,993 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 47362 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:16,993 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7763c91103ebbe310342746ed8aa8aa1/A is initiating minor compaction (all files) 2023-04-13 14:55:16,993 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7763c91103ebbe310342746ed8aa8aa1/B is initiating minor compaction (all files) 2023-04-13 14:55:16,993 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7763c91103ebbe310342746ed8aa8aa1/A in TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,993 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7763c91103ebbe310342746ed8aa8aa1/B in TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:16,993 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/4c35de3c604946a59adcaaff80dc2f21, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/8f9b834f14f84699b4e838f500842749, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/91166ef7c7d74113a74feced3dcfa4de] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp, totalSize=109.3 K 2023-04-13 14:55:16,993 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/ac238953e3924f4cbb9c8054bace2140, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/51e8cb4e0e194422896216d5ded96ecb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/7743183e0d0b43889df07a512fc3e8c4] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp, totalSize=46.3 K 2023-04-13 14:55:16,993 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 4c35de3c604946a59adcaaff80dc2f21, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681397714222 2023-04-13 14:55:16,993 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting ac238953e3924f4cbb9c8054bace2140, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681397714222 2023-04-13 14:55:16,994 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 8f9b834f14f84699b4e838f500842749, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681397714246 2023-04-13 14:55:16,994 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 51e8cb4e0e194422896216d5ded96ecb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681397714246 2023-04-13 14:55:16,994 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 91166ef7c7d74113a74feced3dcfa4de, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681397716473 2023-04-13 14:55:16,994 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 7743183e0d0b43889df07a512fc3e8c4, keycount=400, bloomtype=ROW, size=23.0 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681397716473 2023-04-13 14:55:17,002 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7763c91103ebbe310342746ed8aa8aa1#B#compaction#309 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:17,004 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7763c91103ebbe310342746ed8aa8aa1#A#compaction#308 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:17,066 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:17,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:17,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:17,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:17,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:17,069 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:17,069 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:17,069 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:17,069 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:17,070 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:17,070 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:17,077 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,077 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:17,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:17,078 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,078 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:17,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:17,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:17,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:17,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:17,079 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:17,079 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:17,079 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:17,079 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:17,079 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:17,079 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:17,079 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:17,079 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=147.60 KB heapSize=359.20 KB 2023-04-13 14:55:17,079 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:17,079 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:17,079 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:17,080 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:17,080 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:17,080 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:17,093 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413c59e7bcae3a4417e8d8a761752533bec to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c59e7bcae3a4417e8d8a761752533bec 2023-04-13 14:55:17,098 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=75, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/512e61a72a6741a3a047a8c50f27be17 2023-04-13 14:55:17,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:17,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:17,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397777183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397777183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397777184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397777184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397777185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397777286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397777292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397777293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397777293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397777293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,418 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/7c880a9fd57d4b099f225ab6bb3884cc as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/7c880a9fd57d4b099f225ab6bb3884cc 2023-04-13 14:55:17,423 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7763c91103ebbe310342746ed8aa8aa1/B of 7763c91103ebbe310342746ed8aa8aa1 into 7c880a9fd57d4b099f225ab6bb3884cc(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:17,423 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:17,423 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., storeName=7763c91103ebbe310342746ed8aa8aa1/B, priority=13, startTime=1681397716991; duration=0sec 2023-04-13 14:55:17,423 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:17,423 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:17,424 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 47362 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:17,424 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7763c91103ebbe310342746ed8aa8aa1/C is initiating minor compaction (all files) 2023-04-13 14:55:17,424 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7763c91103ebbe310342746ed8aa8aa1/C in TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:17,424 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/7a20536cb775409ca10915edefa7e410, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/916a970a04444735996ba588d4cbbac1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b295aa232cba4c058c5cdcc39a784d34] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp, totalSize=46.3 K 2023-04-13 14:55:17,424 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 7a20536cb775409ca10915edefa7e410, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681397714222 2023-04-13 14:55:17,425 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 916a970a04444735996ba588d4cbbac1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681397714246 2023-04-13 14:55:17,425 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting b295aa232cba4c058c5cdcc39a784d34, keycount=400, bloomtype=ROW, size=23.0 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681397716473 2023-04-13 14:55:17,429 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7763c91103ebbe310342746ed8aa8aa1#C#compaction#312 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:17,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397777494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397777496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397777496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397777497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397777498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,508 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/6dae7cc1b75e469393d3d18ff5252bca 2023-04-13 14:55:17,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397777801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397777803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397777803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397777804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:17,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397777804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,837 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/59a67e13d77046cf8708833c7adeb2f7 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/59a67e13d77046cf8708833c7adeb2f7 2023-04-13 14:55:17,841 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7763c91103ebbe310342746ed8aa8aa1/C of 7763c91103ebbe310342746ed8aa8aa1 into 59a67e13d77046cf8708833c7adeb2f7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:17,841 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:17,841 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., storeName=7763c91103ebbe310342746ed8aa8aa1/C, priority=13, startTime=1681397716992; duration=0sec 2023-04-13 14:55:17,841 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:17,921 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/c3d35ee93b084a1092f35241e53c47c2 2023-04-13 14:55:17,925 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/512e61a72a6741a3a047a8c50f27be17 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/512e61a72a6741a3a047a8c50f27be17 2023-04-13 14:55:17,928 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/512e61a72a6741a3a047a8c50f27be17, entries=150, sequenceid=75, filesize=25.1 K 2023-04-13 14:55:17,932 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/6dae7cc1b75e469393d3d18ff5252bca as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/6dae7cc1b75e469393d3d18ff5252bca 2023-04-13 14:55:17,935 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/6dae7cc1b75e469393d3d18ff5252bca, entries=150, sequenceid=75, filesize=11.6 K 2023-04-13 14:55:17,938 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/c3d35ee93b084a1092f35241e53c47c2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/c3d35ee93b084a1092f35241e53c47c2 2023-04-13 14:55:17,944 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/c3d35ee93b084a1092f35241e53c47c2, entries=150, sequenceid=75, filesize=11.6 K 2023-04-13 14:55:17,945 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~359.16 KB/367776, currentSize=53.67 KB/54960 for 7763c91103ebbe310342746ed8aa8aa1 in 866ms, sequenceid=75, compaction requested=false 2023-04-13 14:55:17,945 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:17,945 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:17,945 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:17,945 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:17,945 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,945 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:17,946 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:17,947 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,947 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:17,948 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:17,948 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:17,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:17,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@111cd2cd[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:17,950 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:17,951 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:17,953 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:17,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:17,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:17,953 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:17,953 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:17,953 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:17,954 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,954 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:17,955 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,955 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:17,955 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:17,955 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:17,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:17,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:17,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:17,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:17,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:17,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:17,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:17,963 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:17,963 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:17,963 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:17,963 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:17,963 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:17,964 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:17,964 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:17,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:17,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:17,964 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:17,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:17,965 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:17,966 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:17,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,968 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,968 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:17,968 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:17,968 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:17,968 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:17,968 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:17,968 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:17,968 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:17,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:17,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:17,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:17,969 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:17,969 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:17,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:17,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:17,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:17,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:18,069 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:18,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:18,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:18,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:18,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:18,072 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:18,072 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:18,072 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:18,072 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:18,072 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:18,072 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:18,079 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,079 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:18,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:18,079 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,079 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:18,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:18,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:18,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:18,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:18,080 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:18,080 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:18,080 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:18,081 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:18,081 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:18,081 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:18,081 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:18,081 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 14:55:18,081 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:18,081 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:18,081 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:18,081 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:18,081 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:18,081 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:18,227 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/e1c0c06953c348e18786140e69952c18 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/e1c0c06953c348e18786140e69952c18 2023-04-13 14:55:18,231 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7763c91103ebbe310342746ed8aa8aa1/A of 7763c91103ebbe310342746ed8aa8aa1 into e1c0c06953c348e18786140e69952c18(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-13 14:55:18,231 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:18,231 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., storeName=7763c91103ebbe310342746ed8aa8aa1/A, priority=13, startTime=1681397716991; duration=1sec 2023-04-13 14:55:18,232 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:18,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:18,314 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:18,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397778334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397778334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397718320, max timestamp=1681397718340Num uniques 859; 2023-04-13 14:55:18,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:18,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397718320, max timestamp=1681397718340Num uniques 859; 2023-04-13 14:55:18,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:18,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397718320, max timestamp=1681397718340Num uniques 859; 2023-04-13 14:55:18,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:18,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397778372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397778373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397778379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397778442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397778443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397778480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397778480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397778486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,501 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413a963442361164db5b075571fa76ea78a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413a963442361164db5b075571fa76ea78a 2023-04-13 14:55:18,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397778645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397778650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397778686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397778687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397778696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,909 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=88, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/2d588b28c7524ad5a78dfbd8b899b324 2023-04-13 14:55:18,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397778952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397778956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:18,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:18,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397778994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:19,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397778994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:19,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397779001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,332 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=88 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/8a9ab894950d428ab649ffc1f8146982 2023-04-13 14:55:19,353 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=88 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/bd9e1311412d4872948d39272fb08c71 2023-04-13 14:55:19,360 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/2d588b28c7524ad5a78dfbd8b899b324 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/2d588b28c7524ad5a78dfbd8b899b324 2023-04-13 14:55:19,367 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/2d588b28c7524ad5a78dfbd8b899b324, entries=150, sequenceid=88, filesize=25.1 K 2023-04-13 14:55:19,367 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/8a9ab894950d428ab649ffc1f8146982 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/8a9ab894950d428ab649ffc1f8146982 2023-04-13 14:55:19,377 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/8a9ab894950d428ab649ffc1f8146982, entries=150, sequenceid=88, filesize=11.6 K 2023-04-13 14:55:19,378 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/bd9e1311412d4872948d39272fb08c71 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/bd9e1311412d4872948d39272fb08c71 2023-04-13 14:55:19,384 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/bd9e1311412d4872948d39272fb08c71, entries=150, sequenceid=88, filesize=11.6 K 2023-04-13 14:55:19,385 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 7763c91103ebbe310342746ed8aa8aa1 in 1304ms, sequenceid=88, compaction requested=true 2023-04-13 14:55:19,387 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:19,390 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:19,390 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:19,390 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:19,390 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,390 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:19,390 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:19,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:19,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:19,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:19,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@788cbd0a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:19,393 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:19,394 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,395 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,396 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:19,396 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:19,396 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:19,396 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,396 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:19,397 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,397 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:19,397 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:19,397 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:19,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:19,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:19,400 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:19,400 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:19,400 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:19,400 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:19,400 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:19,402 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:19,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:19,403 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,404 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:19,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,408 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,408 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:19,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:19,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:19,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:19,408 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:19,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:19,409 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:19,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,409 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:19,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:19,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:19,466 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-13 14:55:19,466 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:19,466 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:19,466 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:19,466 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:19,466 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:19,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:19,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:19,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397779494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:19,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397779495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:19,509 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:19,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397779503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:19,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397779503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:19,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:19,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:19,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:19,519 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:19,519 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:19,519 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:19,519 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:19,519 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,519 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:19,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397779510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,520 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:19,520 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:19,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:19,521 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,521 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:19,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:19,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:19,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:19,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:19,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:19,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:19,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:19,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:19,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:19,522 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:19,522 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:19,522 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:19,522 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:19,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:19,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:19,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:19,522 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:19,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:19,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:19,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eac0922[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:19,527 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:19,527 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,528 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:19,529 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,529 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:19,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:19,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:19,531 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,531 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:19,531 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:19,531 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:19,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:19,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:19,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:19,533 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:19,533 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:19,533 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:19,533 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:19,533 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:19,535 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,535 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:19,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,536 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:19,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:19,536 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:19,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:19,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:19,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:19,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:19,542 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,542 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,542 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:19,542 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,543 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,543 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,543 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,543 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:19,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:19,543 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:19,543 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:19,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:19,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:19,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397779598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:19,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397779601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,643 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:19,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:19,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:19,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:19,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:19,649 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:19,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:19,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:19,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:19,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,651 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:19,651 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:19,652 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,652 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:19,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:19,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:19,653 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:19,653 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:19,653 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:19,653 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:19,653 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:19,653 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:19,653 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:19,653 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:19,653 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:19,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:19,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:19,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:19,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:19,658 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,659 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:19,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:19,672 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,672 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:19,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:19,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:19,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d8b54c8[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:19,674 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:19,674 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,675 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,675 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:19,675 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:19,675 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:19,675 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,675 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:19,676 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:19,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:19,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:19,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:19,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:19,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:19,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:19,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:19,679 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:19,679 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:19,679 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:19,679 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:19,679 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:19,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:19,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:19,681 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,685 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,687 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:19,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:19,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:19,687 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:19,687 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:19,687 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:19,687 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:19,687 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:19,687 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:19,687 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,687 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,688 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,688 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,688 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,787 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:19,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:19,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:19,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:19,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:19,791 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:19,791 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:19,791 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:19,791 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:19,791 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,791 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,793 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:19,793 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:19,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,794 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,794 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:19,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:19,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:19,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:19,795 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:19,795 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:19,795 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:19,795 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:19,795 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:19,795 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:19,795 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:19,796 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:19,796 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:19,796 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:19,796 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:19,796 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,796 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:19,796 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,797 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,798 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,798 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:19,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:19,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:19,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a76770e[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:19,807 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:19,807 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,809 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,809 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:19,809 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:19,809 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:19,809 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,810 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:19,811 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:19,811 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:19,811 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:19,811 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:19,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:19,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:19,814 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:19,814 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:19,814 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:19,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:19,814 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:19,814 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:19,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397779809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:19,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397779812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,818 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,818 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:19,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,818 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:19,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:19,818 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,820 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,822 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,822 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,822 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,822 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:19,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:19,822 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:19,822 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,822 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:19,822 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,822 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:19,822 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:19,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:19,822 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:19,822 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:19,886 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304131a023faf0bce4de98b4ff01984fd3749 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304131a023faf0bce4de98b4ff01984fd3749 2023-04-13 14:55:19,922 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:19,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:19,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:19,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:19,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:19,926 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:19,926 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:19,926 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:19,926 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:19,926 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,926 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,928 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,928 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:19,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:19,929 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,929 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:19,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:19,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:19,930 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:19,930 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:19,931 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:19,931 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:19,931 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:19,931 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:19,931 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,931 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:19,931 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:19,932 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,932 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:19,932 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,933 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:19,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:19,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@770fe3a7[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:19,934 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:19,934 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,935 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,935 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:19,935 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,935 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:19,935 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:19,935 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:19,936 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:19,938 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,938 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:19,938 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:19,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:19,940 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:19,940 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:19,940 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:19,940 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:19,940 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:19,943 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,943 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,943 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:19,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:19,944 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:19,946 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,949 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,949 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,949 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:19,949 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:19,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:19,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:19,949 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:19,949 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:19,949 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,949 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:19,949 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:19,949 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:19,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:19,949 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:19,949 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:19,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:20,050 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:20,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:20,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:20,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:20,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:20,053 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:20,053 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:20,053 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:20,053 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:20,053 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,053 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,056 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,056 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:20,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:20,056 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,056 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:20,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:20,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:20,057 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:20,057 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:20,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:20,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:20,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:20,058 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:20,058 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:20,058 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:20,058 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:20,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:20,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:20,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:20,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:20,060 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,060 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,060 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,060 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:20,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f4017d[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:20,063 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:20,063 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,064 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,064 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:20,064 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:20,064 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:20,065 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,065 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:20,070 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,070 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:20,070 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:20,070 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:20,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:20,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:20,073 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:20,073 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:20,073 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:20,073 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:20,073 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:20,076 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,076 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,076 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:20,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,077 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,079 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,081 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,081 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:20,081 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:20,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:20,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:20,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:20,081 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,082 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,082 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,082 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,082 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:20,082 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,082 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:20,081 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:20,082 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:20,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:20,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:20,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397780116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:20,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397780117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,182 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:20,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:20,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:20,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:20,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:20,185 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:20,185 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:20,185 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:20,185 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:20,186 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,186 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,188 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,188 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:20,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:20,189 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,189 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:20,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:20,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:20,190 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:20,190 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:20,190 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:20,190 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:20,190 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:20,190 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:20,191 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:20,191 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:20,191 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:20,191 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:20,191 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:20,191 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,191 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:20,191 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:20,192 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,193 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,193 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,193 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:20,194 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:20,195 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e1e259f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:20,196 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,196 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:20,196 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:20,196 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:20,196 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,196 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:20,197 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,197 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:20,197 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,197 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:20,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:20,202 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:20,202 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:20,202 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:20,202 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:20,202 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:20,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,206 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:20,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:20,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,208 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,210 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,210 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:20,210 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:20,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:20,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:20,210 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:20,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:20,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:20,211 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:20,211 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:20,210 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:20,211 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,211 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,211 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,211 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,211 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,293 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=125, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/e1de50f0883346628dc8457e547f299e 2023-04-13 14:55:20,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/e68eee004d0941f391429a2b355f1281 2023-04-13 14:55:20,311 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:20,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:20,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:20,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:20,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:20,314 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:20,314 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:20,314 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:20,314 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:20,314 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,314 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,315 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,315 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:20,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:20,316 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,316 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:20,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:20,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:20,317 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:20,317 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:20,317 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:20,317 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:20,317 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:20,319 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:20,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7789e955[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:20,320 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:20,320 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,322 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,322 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:20,322 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:20,323 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:20,323 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,323 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:20,324 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:20,324 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,324 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,324 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:20,326 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:20,326 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:20,326 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:20,326 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:20,326 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:20,328 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,328 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,328 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:20,328 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,330 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,332 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,332 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:20,332 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:20,332 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:20,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:20,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:20,332 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:20,332 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:20,332 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:20,332 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,332 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:20,332 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,332 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,332 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,432 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:20,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:20,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:20,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:20,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:20,435 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:20,435 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:20,436 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:20,436 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:20,436 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,436 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,438 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,438 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:20,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:20,438 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,438 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:20,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:20,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:20,440 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:20,440 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:20,440 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:20,440 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:20,442 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,442 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,442 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,442 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f132ed8[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:20,443 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:20,443 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,444 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,444 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,444 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:20,445 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:20,445 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:20,445 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:20,446 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,446 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:20,446 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:20,446 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:20,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:20,448 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:20,448 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:20,448 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:20,448 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:20,448 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:20,451 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,451 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,451 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:20,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,452 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,453 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,457 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:20,457 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:20,457 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,457 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:20,457 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:20,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:20,457 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:20,457 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:20,457 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:20,457 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,457 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,457 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:20,458 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:20,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397780514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:20,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397780515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:20,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397780527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,558 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:20,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:20,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:20,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:20,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:20,561 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:20,561 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:20,561 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:20,561 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:20,561 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,561 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,563 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,563 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:20,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:20,563 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,563 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:20,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:20,564 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:20,564 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:20,564 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:20,564 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:20,564 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:20,564 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:20,564 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:20,564 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:20,565 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:20,565 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:20,565 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:20,565 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,565 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:20,565 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:20,567 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,567 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,568 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,568 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:20,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36a527ae[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:20,569 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:20,569 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,570 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,571 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:20,571 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,571 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:20,571 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:20,571 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:20,572 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:20,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:20,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:20,574 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:20,574 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:20,574 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:20,574 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:20,574 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:20,576 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,576 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,576 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:20,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:20,576 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,577 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,581 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,581 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:20,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:20,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:20,581 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:20,581 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:20,581 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,581 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:20,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:20,582 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:20,581 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,582 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,582 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,582 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,582 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:20,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:20,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397780624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:20,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397780626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,682 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:20,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:20,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:20,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:20,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:20,685 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:20,685 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:20,685 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:20,685 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:20,686 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,686 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,687 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:20,687 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:20,687 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,687 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:20,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:20,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:20,688 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:20,689 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:20,689 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:20,689 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:20,689 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:20,691 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,691 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,692 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,692 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6948de38[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:20,693 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:20,694 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,695 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:20,695 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,695 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:20,696 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:20,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:20,699 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:20,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:20,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:20,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:20,699 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:20,700 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,700 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:20,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,701 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,701 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:20,703 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,705 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:20,705 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:20,705 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:20,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:20,706 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:20,706 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:20,705 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:20,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:20,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:20,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:20,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:20,712 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/c5f1a76cad064a5bb5195b7fbd81585a 2023-04-13 14:55:20,716 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/e1de50f0883346628dc8457e547f299e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/e1de50f0883346628dc8457e547f299e 2023-04-13 14:55:20,719 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/e1de50f0883346628dc8457e547f299e, entries=200, sequenceid=125, filesize=31.9 K 2023-04-13 14:55:20,720 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/e68eee004d0941f391429a2b355f1281 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/e68eee004d0941f391429a2b355f1281 2023-04-13 14:55:20,727 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/e68eee004d0941f391429a2b355f1281, entries=150, sequenceid=125, filesize=11.6 K 2023-04-13 14:55:20,727 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/c5f1a76cad064a5bb5195b7fbd81585a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/c5f1a76cad064a5bb5195b7fbd81585a 2023-04-13 14:55:20,733 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/c5f1a76cad064a5bb5195b7fbd81585a, entries=150, sequenceid=125, filesize=11.6 K 2023-04-13 14:55:20,734 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 7763c91103ebbe310342746ed8aa8aa1 in 1267ms, sequenceid=125, compaction requested=true 2023-04-13 14:55:20,734 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:20,734 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:20,734 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:55:20,734 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:55:20,734 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:20,734 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:20,739 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110027 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:55:20,739 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:55:20,739 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7763c91103ebbe310342746ed8aa8aa1/B is initiating minor compaction (all files) 2023-04-13 14:55:20,739 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7763c91103ebbe310342746ed8aa8aa1/B in TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:20,739 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/7c880a9fd57d4b099f225ab6bb3884cc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/6dae7cc1b75e469393d3d18ff5252bca, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/8a9ab894950d428ab649ffc1f8146982, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/e68eee004d0941f391429a2b355f1281] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp, totalSize=46.5 K 2023-04-13 14:55:20,739 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7763c91103ebbe310342746ed8aa8aa1/A is initiating minor compaction (all files) 2023-04-13 14:55:20,739 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7763c91103ebbe310342746ed8aa8aa1/A in TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:20,740 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/e1c0c06953c348e18786140e69952c18, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/512e61a72a6741a3a047a8c50f27be17, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/2d588b28c7524ad5a78dfbd8b899b324, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/e1de50f0883346628dc8457e547f299e] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp, totalSize=107.4 K 2023-04-13 14:55:20,741 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 7c880a9fd57d4b099f225ab6bb3884cc, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681397716477 2023-04-13 14:55:20,741 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting e1c0c06953c348e18786140e69952c18, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681397716477 2023-04-13 14:55:20,741 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 6dae7cc1b75e469393d3d18ff5252bca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681397716552 2023-04-13 14:55:20,741 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 512e61a72a6741a3a047a8c50f27be17, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681397716552 2023-04-13 14:55:20,742 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 2d588b28c7524ad5a78dfbd8b899b324, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1681397717183 2023-04-13 14:55:20,742 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 8a9ab894950d428ab649ffc1f8146982, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1681397717183 2023-04-13 14:55:20,742 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting e1de50f0883346628dc8457e547f299e, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681397718355 2023-04-13 14:55:20,742 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting e68eee004d0941f391429a2b355f1281, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681397718355 2023-04-13 14:55:20,750 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7763c91103ebbe310342746ed8aa8aa1#B#compaction#320 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:20,753 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7763c91103ebbe310342746ed8aa8aa1#A#compaction#321 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:20,806 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:20,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:20,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:20,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:20,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:20,810 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:20,810 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:20,810 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:20,810 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:20,810 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,810 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,811 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:20,811 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:20,812 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:20,812 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:20,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:20,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:20,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:20,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:20,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:20,813 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:20,813 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:20,813 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:20,813 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:20,813 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:20,813 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:20,813 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:20,814 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:55:20,814 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:20,814 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:20,814 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:20,814 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:20,814 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:20,814 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:20,867 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041313d55ebf919f4bd5944d9a5185d4c2b4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041313d55ebf919f4bd5944d9a5185d4c2b4 2023-04-13 14:55:20,884 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=135, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/a27b06e47b34446ba704957043008c9e 2023-04-13 14:55:20,926 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/48dc3c99f77d4b488dcf26bee09c01fc 2023-04-13 14:55:21,168 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/134e4d6db0ae45a88640a3b2724984a2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/134e4d6db0ae45a88640a3b2724984a2 2023-04-13 14:55:21,173 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7763c91103ebbe310342746ed8aa8aa1/B of 7763c91103ebbe310342746ed8aa8aa1 into 134e4d6db0ae45a88640a3b2724984a2(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:21,173 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:21,173 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., storeName=7763c91103ebbe310342746ed8aa8aa1/B, priority=12, startTime=1681397720734; duration=0sec 2023-04-13 14:55:21,173 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:21,173 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:55:21,175 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:55:21,175 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7763c91103ebbe310342746ed8aa8aa1/C is initiating minor compaction (all files) 2023-04-13 14:55:21,175 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7763c91103ebbe310342746ed8aa8aa1/C in TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:21,175 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/59a67e13d77046cf8708833c7adeb2f7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/c3d35ee93b084a1092f35241e53c47c2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/bd9e1311412d4872948d39272fb08c71, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/c5f1a76cad064a5bb5195b7fbd81585a] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp, totalSize=46.5 K 2023-04-13 14:55:21,175 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 59a67e13d77046cf8708833c7adeb2f7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681397716477 2023-04-13 14:55:21,176 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting c3d35ee93b084a1092f35241e53c47c2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681397716552 2023-04-13 14:55:21,176 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting bd9e1311412d4872948d39272fb08c71, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1681397717183 2023-04-13 14:55:21,176 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting c5f1a76cad064a5bb5195b7fbd81585a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681397718355 2023-04-13 14:55:21,192 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7763c91103ebbe310342746ed8aa8aa1#C#compaction#325 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:21,337 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/abca699ce6c74e63b02c11b7b0f890e5 2023-04-13 14:55:21,342 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/a27b06e47b34446ba704957043008c9e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/a27b06e47b34446ba704957043008c9e 2023-04-13 14:55:21,347 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/a27b06e47b34446ba704957043008c9e, entries=100, sequenceid=135, filesize=18.5 K 2023-04-13 14:55:21,348 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/48dc3c99f77d4b488dcf26bee09c01fc as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/48dc3c99f77d4b488dcf26bee09c01fc 2023-04-13 14:55:21,352 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/48dc3c99f77d4b488dcf26bee09c01fc, entries=100, sequenceid=135, filesize=9.4 K 2023-04-13 14:55:21,353 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/abca699ce6c74e63b02c11b7b0f890e5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/abca699ce6c74e63b02c11b7b0f890e5 2023-04-13 14:55:21,358 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/abca699ce6c74e63b02c11b7b0f890e5, entries=100, sequenceid=135, filesize=9.4 K 2023-04-13 14:55:21,359 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 7763c91103ebbe310342746ed8aa8aa1 in 545ms, sequenceid=135, compaction requested=false 2023-04-13 14:55:21,359 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:21,359 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:21,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:21,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:21,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:21,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:21,361 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,361 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:21,361 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,361 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51b35105[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:21,363 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:21,364 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,366 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,366 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:21,366 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:21,366 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:21,366 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,366 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:21,367 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,367 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:21,367 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:21,367 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:21,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:21,370 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:21,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:21,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:21,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:21,370 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:21,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:21,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:21,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,376 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:21,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,377 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,393 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,394 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:21,394 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:21,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:21,393 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:21,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:21,394 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,394 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:21,394 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:21,394 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:21,394 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,394 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:21,394 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:21,394 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:21,494 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:21,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:21,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:21,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:21,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:21,500 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:21,500 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:21,500 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:21,500 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:21,500 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,500 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:21,502 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,502 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:21,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:21,502 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,502 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:21,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:21,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:21,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:21,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:21,504 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:21,504 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:21,504 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:21,504 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:21,504 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:21,504 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:21,504 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:21,505 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:21,505 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:21,505 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:21,505 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:21,505 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,505 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:21,505 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,506 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:21,507 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,507 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:21,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:21,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37b0d005[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:21,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:21,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,510 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:21,511 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,511 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:21,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:21,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:21,513 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:21,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:21,516 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:21,516 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:21,516 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:21,516 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:21,516 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:21,518 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,518 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:21,518 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:21,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:21,519 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,521 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,523 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:21,523 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:21,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:21,523 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,524 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:21,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:21,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:21,524 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:21,524 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:21,524 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,524 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,524 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,524 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:21,524 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:21,524 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:21,620 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/8f603c721fdb402a85c49ff4bc3239fc as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/8f603c721fdb402a85c49ff4bc3239fc 2023-04-13 14:55:21,624 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:21,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:21,627 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7763c91103ebbe310342746ed8aa8aa1/C of 7763c91103ebbe310342746ed8aa8aa1 into 8f603c721fdb402a85c49ff4bc3239fc(size=11.8 K), total size for store is 21.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:21,627 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:21,627 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., storeName=7763c91103ebbe310342746ed8aa8aa1/C, priority=12, startTime=1681397720734; duration=0sec 2023-04-13 14:55:21,627 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:21,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:21,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:21,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:21,636 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:21,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:21,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:21,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:21,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,637 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:21,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:21,638 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:21,638 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,638 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:21,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:21,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:21,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:21,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:21,640 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:21,640 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:21,640 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:21,640 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:21,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:21,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:21,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:21,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:21,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:21,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:21,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:21,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:21,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:21,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,644 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:21,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:21,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28b7190a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:21,647 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:21,647 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,649 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:21,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:21,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:21,656 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,656 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:21,657 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:21,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:21,659 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:21,660 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:21,660 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:21,660 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:21,660 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:21,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:21,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:21,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:21,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:21,662 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,663 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,666 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:21,666 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:21,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:21,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:21,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:21,666 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:21,666 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:21,666 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:21,666 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:21,666 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:21,666 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:21,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:21,667 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,667 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,667 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:21,667 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,767 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:21,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:21,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:21,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:21,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:21,773 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:21,773 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:21,773 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:21,773 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:21,773 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,773 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:21,774 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:21,774 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:21,775 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:21,775 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:21,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:21,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:21,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:21,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:21,776 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:21,776 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:21,776 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 14:55:21,776 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:21,777 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:21,777 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:21,777 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:21,777 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:21,777 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:21,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:21,828 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:21,857 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413e4475d20341543daa07b417ff254e957 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e4475d20341543daa07b417ff254e957 2023-04-13 14:55:21,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397721868, max timestamp=1681397721906Num uniques 859; 2023-04-13 14:55:21,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397721868, max timestamp=1681397721906Num uniques 859; 2023-04-13 14:55:21,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397721868, max timestamp=1681397721906Num uniques 859; 2023-04-13 14:55:21,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:21,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:21,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:22,006 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/cd235594f022426fb740bac411811466 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/cd235594f022426fb740bac411811466 2023-04-13 14:55:22,011 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7763c91103ebbe310342746ed8aa8aa1/A of 7763c91103ebbe310342746ed8aa8aa1 into cd235594f022426fb740bac411811466(size=25.4 K), total size for store is 43.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-13 14:55:22,011 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:22,011 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., storeName=7763c91103ebbe310342746ed8aa8aa1/A, priority=12, startTime=1681397720734; duration=1sec 2023-04-13 14:55:22,011 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:22,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681397721982, max timestamp=1681397722017Num uniques 1018; 2023-04-13 14:55:22,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-13 14:55:22,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681397721982, max timestamp=1681397722017Num uniques 1018; 2023-04-13 14:55:22,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-13 14:55:22,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681397721982, max timestamp=1681397722017Num uniques 1018; 2023-04-13 14:55:22,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-13 14:55:22,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:22,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397782086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:22,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397782093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:22,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397782194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,210 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:22,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397782204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,278 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=148, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/28791ce94b4b4d68bd29b06bf734d01d 2023-04-13 14:55:22,301 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/5189738cd7844f0299ac2cdf71c019d6 2023-04-13 14:55:22,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:22,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397782403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:22,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397782412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:22,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397782532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,535 DEBUG [Thread-2236] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:55:22,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:22,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397782533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,542 DEBUG [Thread-2238] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:55:22,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:22,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397782538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,544 DEBUG [Thread-2242] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:55:22,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:22,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397782708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:22,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397782716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,723 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/430e970922da4c659e0094096c1e5ccd 2023-04-13 14:55:22,727 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/28791ce94b4b4d68bd29b06bf734d01d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/28791ce94b4b4d68bd29b06bf734d01d 2023-04-13 14:55:22,731 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/28791ce94b4b4d68bd29b06bf734d01d, entries=200, sequenceid=148, filesize=32.1 K 2023-04-13 14:55:22,732 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/5189738cd7844f0299ac2cdf71c019d6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/5189738cd7844f0299ac2cdf71c019d6 2023-04-13 14:55:22,735 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/5189738cd7844f0299ac2cdf71c019d6, entries=150, sequenceid=148, filesize=11.8 K 2023-04-13 14:55:22,735 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/430e970922da4c659e0094096c1e5ccd as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/430e970922da4c659e0094096c1e5ccd 2023-04-13 14:55:22,738 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/430e970922da4c659e0094096c1e5ccd, entries=150, sequenceid=148, filesize=11.8 K 2023-04-13 14:55:22,739 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 7763c91103ebbe310342746ed8aa8aa1 in 963ms, sequenceid=148, compaction requested=true 2023-04-13 14:55:22,739 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:22,739 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:22,739 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:22,739 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:22,739 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,739 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:22,739 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:22,741 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,741 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:22,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:22,743 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:22,743 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:22,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:22,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:22,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:22,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:22,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@721ab3a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:22,744 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:22,744 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:22,745 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:22,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:22,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:22,745 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:22,745 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:22,745 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,745 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:22,745 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:22,746 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:22,746 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,746 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:22,746 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:22,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:22,748 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:22,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:22,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:22,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:22,748 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:22,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:22,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:22,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:22,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:22,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:22,753 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:22,755 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,757 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,757 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:22,757 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:22,757 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:22,757 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:22,757 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:22,757 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,757 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:22,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:22,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:22,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:22,757 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:22,758 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:22,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:22,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:22,758 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:22,758 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:22,858 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:22,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:22,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:22,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:22,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:22,862 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:22,862 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:22,862 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:22,862 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:22,863 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:22,863 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:22,867 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:22,867 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:22,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:22,868 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:22,868 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:22,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:22,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:22,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:22,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:22,869 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:22,869 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:22,869 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:22,869 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:22,869 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:22,871 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:22,871 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:22,871 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=140.89 KB heapSize=399.38 KB 2023-04-13 14:55:22,871 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:22,871 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:22,871 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:22,871 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:22,871 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:22,871 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:23,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:23,221 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:23,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:23,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397783231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:23,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397783240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,284 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304137cb065b4a4ca4edd81fa7bf7cf33624c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137cb065b4a4ca4edd81fa7bf7cf33624c 2023-04-13 14:55:23,289 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=203, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/ca8477f869da4aabbcd0f970ce834111 2023-04-13 14:55:23,298 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/de3f25d68f73496d8208d8e796112589 2023-04-13 14:55:23,307 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/ff03f30cd5794974b9982ccb9c767562 2023-04-13 14:55:23,310 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/ca8477f869da4aabbcd0f970ce834111 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/ca8477f869da4aabbcd0f970ce834111 2023-04-13 14:55:23,314 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/ca8477f869da4aabbcd0f970ce834111, entries=150, sequenceid=203, filesize=25.3 K 2023-04-13 14:55:23,314 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/de3f25d68f73496d8208d8e796112589 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/de3f25d68f73496d8208d8e796112589 2023-04-13 14:55:23,317 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/de3f25d68f73496d8208d8e796112589, entries=150, sequenceid=203, filesize=11.8 K 2023-04-13 14:55:23,318 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/ff03f30cd5794974b9982ccb9c767562 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/ff03f30cd5794974b9982ccb9c767562 2023-04-13 14:55:23,321 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/ff03f30cd5794974b9982ccb9c767562, entries=150, sequenceid=203, filesize=11.8 K 2023-04-13 14:55:23,322 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~399.33 KB/408912, currentSize=40.25 KB/41220 for 7763c91103ebbe310342746ed8aa8aa1 in 451ms, sequenceid=203, compaction requested=true 2023-04-13 14:55:23,322 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:23,322 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:23,322 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:23,322 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:23,322 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,322 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:23,322 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:23,324 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,324 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,324 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,324 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:23,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd1f9a2[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:23,326 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:23,326 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,327 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,327 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:23,327 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:23,327 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:23,327 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,327 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:23,330 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:23,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:23,332 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:23,332 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:23,332 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:23,332 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:23,332 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:23,335 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,335 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,335 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,335 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,336 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,339 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:23,339 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:23,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:23,339 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:23,340 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:23,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:23,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:23,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:23,339 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,340 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,340 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,340 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,340 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,340 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,340 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:23,340 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:23,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:23,345 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:55:23,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:23,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:23,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:23,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:23,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:23,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:23,356 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304134ca8fe855db64651b26d79c8498e073f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134ca8fe855db64651b26d79c8498e073f 2023-04-13 14:55:23,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397723404, max timestamp=1681397723425Num uniques 859; 2023-04-13 14:55:23,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:23,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397723404, max timestamp=1681397723425Num uniques 859; 2023-04-13 14:55:23,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:23,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397723404, max timestamp=1681397723425Num uniques 859; 2023-04-13 14:55:23,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:23,440 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:23,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:23,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:23,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:23,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:23,443 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:23,443 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:23,443 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:23,443 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:23,444 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,444 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,445 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:23,445 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:23,445 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,445 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:23,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:23,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:23,446 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:23,446 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:23,446 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:23,447 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:23,447 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:23,447 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:23,447 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:23,447 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:23,447 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:23,447 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:23,447 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:23,447 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,447 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:23,447 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:23,448 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,448 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,449 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,449 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:23,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d5038c5[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:23,451 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:23,451 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,453 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,453 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:23,453 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:23,453 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:23,453 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,453 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:23,454 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:23,454 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:23,454 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,454 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:23,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:23,457 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:23,457 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:23,457 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:23,457 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:23,457 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:23,458 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,458 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,458 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:23,459 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,460 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:23,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:23,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:23,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:23,462 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:23,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:23,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:23,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,463 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:23,463 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:23,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:23,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:23,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397783484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:23,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397783486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,563 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:23,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:23,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:23,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:23,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:23,566 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:23,566 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:23,566 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:23,566 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:23,566 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,567 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,568 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:23,568 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:23,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:23,569 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,569 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:23,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:23,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:23,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:23,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:23,571 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:23,571 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:23,571 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:23,571 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:23,571 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:23,571 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:23,571 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:23,571 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:23,571 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:23,571 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:23,571 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:23,571 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,571 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:23,571 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:23,574 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,574 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,575 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,575 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:23,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:23,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d3f112c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:23,576 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:23,576 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,577 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,577 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:23,577 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:23,577 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,577 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:23,577 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:23,578 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,578 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:23,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,579 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:23,579 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:23,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:23,581 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:23,581 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:23,581 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:23,581 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:23,581 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:23,584 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,584 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:23,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,584 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:23,585 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,586 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,587 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:23,587 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:23,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:23,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:23,588 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:23,588 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:23,588 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:23,588 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:23,588 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,588 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:23,588 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,588 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,588 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,588 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:23,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:23,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57312 deadline: 1681397783593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:23,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397783593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,688 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:23,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:23,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:23,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:23,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:23,691 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:23,691 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:23,691 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:23,691 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:23,692 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,692 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,693 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:23,693 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:23,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:23,693 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,693 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:23,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:23,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:23,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:23,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:23,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:23,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:23,700 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:23,700 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:23,700 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:23,700 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:23,700 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:23,700 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:23,700 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:23,700 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:23,700 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,700 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:23,700 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:23,708 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,708 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,712 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,712 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:23,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67cd0fca[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:23,713 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:23,713 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,715 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,715 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,715 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:23,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:23,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,717 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:23,717 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:23,717 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:23,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:23,719 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:23,719 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:23,719 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:23,719 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:23,719 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:23,722 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,722 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,722 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:23,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,722 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,725 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:23,732 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:23,732 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:23,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:23,732 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:23,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:23,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:23,732 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,732 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,732 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,732 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,732 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:23,732 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:23,732 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:23,733 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,733 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,761 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=213, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/3ecfed8f6ffe48c9a1ef0ac8507b4a41 2023-04-13 14:55:23,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:23,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397783808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:23,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57344 deadline: 1681397783808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,833 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:23,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:23,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:23,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:23,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:23,836 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:23,836 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:23,836 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:23,836 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:23,837 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,837 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,838 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:23,838 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,839 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,839 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:23,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:23,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:23,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:23,840 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:23,841 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:23,841 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:23,841 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:23,842 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,842 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,842 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,843 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:23,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2412f678[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:23,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:23,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,846 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:23,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:23,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:23,846 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,846 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:23,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:23,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:23,847 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,848 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:23,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:23,850 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:23,850 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:23,850 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:23,850 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:23,850 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:23,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:23,852 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,853 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,855 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,855 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,855 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,855 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:23,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:23,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:23,855 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:23,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:23,855 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:23,855 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:23,855 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:23,855 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:23,855 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:23,855 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,855 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,955 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:23,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:23,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:23,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:23,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:23,958 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:23,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:23,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:23,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:23,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,961 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,961 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:23,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:23,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:23,961 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,961 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:23,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:23,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:23,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:23,962 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:23,962 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:23,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:23,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:23,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:23,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:23,964 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,965 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,965 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:23,966 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:23,966 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58f8642f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:23,967 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:23,968 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,968 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:23,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:23,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:23,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:23,971 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:23,971 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:23,971 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:23,971 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:23,971 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:23,973 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,973 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,973 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:23,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,974 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:23,976 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:23,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:23,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:23,979 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:23,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:23,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:23,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:23,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:23,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:23,980 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,980 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:23,980 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:23,982 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:23,982 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:23,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:23,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:24,082 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:24,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:24,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:24,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:24,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:24,085 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:24,085 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:24,085 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:24,085 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:24,085 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,085 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,087 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,087 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:24,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:24,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:24,087 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,087 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:24,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:24,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:24,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:24,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:24,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:24,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:24,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:24,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:24,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:24,088 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,088 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:24,088 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:24,089 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,089 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:24,089 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:24,089 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,089 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:24,089 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:24,090 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,090 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,090 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,090 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:24,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d5bb86d[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:24,092 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:24,092 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,094 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,094 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:24,094 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:24,094 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:24,094 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,094 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:24,103 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:24,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:24,105 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:24,105 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:24,105 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:24,105 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:24,105 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:24,106 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,106 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,107 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:24,107 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,108 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,110 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:24,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:24,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:24,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:24,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:24,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:24,111 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:24,111 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,111 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:24,111 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:24,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:24,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:24,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397784116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397784116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/86536308cd6541f295cadd068b196f32 2023-04-13 14:55:24,211 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:24,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:24,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:24,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:24,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:24,214 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:24,214 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:24,214 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:24,214 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:24,215 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,215 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,216 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,216 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:24,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:24,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:24,217 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,217 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:24,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:24,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:24,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:24,218 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,218 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:24,218 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:24,218 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:24,219 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,219 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:24,219 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:24,220 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,220 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,220 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,220 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:24,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42e74d53[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:24,221 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:24,221 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,222 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,222 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:24,223 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,223 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:24,223 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:24,223 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:24,224 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:24,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:24,226 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:24,226 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:24,226 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:24,226 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:24,226 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:24,228 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,228 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,229 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:24,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,229 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,230 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,232 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,232 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,232 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:24,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:24,232 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,232 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:24,232 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,232 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:24,232 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,233 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:24,233 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:24,234 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:24,234 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:24,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:24,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:24,334 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:24,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:24,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:24,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:24,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:24,337 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:24,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:24,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:24,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:24,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,338 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:24,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:24,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:24,339 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,339 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:24,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:24,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:24,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:24,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:24,340 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:24,340 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:24,340 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:24,340 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:24,340 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:24,340 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,340 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:24,340 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:24,340 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,340 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:24,340 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:24,340 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,340 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:24,341 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:24,342 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,342 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,342 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,342 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:24,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7175c241[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:24,343 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:24,344 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,345 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:24,347 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:24,347 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:24,347 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,347 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:24,347 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:24,347 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:24,347 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:24,347 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:24,347 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:24,347 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:24,347 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:24,349 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,349 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,349 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:24,349 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,350 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,352 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,352 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,352 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,352 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:24,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:24,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:24,352 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:24,352 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:24,352 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,353 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,353 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,353 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:24,353 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:24,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:24,353 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:24,453 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:24,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:24,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:24,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:24,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:24,457 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:24,457 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:24,457 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:24,457 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:24,458 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,458 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,460 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:24,460 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:24,460 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,460 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:24,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:24,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:24,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:24,461 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:24,461 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:24,462 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:24,462 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:24,462 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:24,462 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,462 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:24,462 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:24,462 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,462 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:24,462 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:24,462 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,462 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:24,462 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:24,464 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,464 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,464 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,464 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:24,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ab7195c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:24,466 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:24,466 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,467 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,467 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,468 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:24,467 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:24,469 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:24,469 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:24,469 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:24,470 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:24,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:24,472 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:24,473 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:24,473 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:24,473 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:24,473 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:24,479 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,479 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:24,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,479 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:24,480 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,481 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,486 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,486 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:24,486 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:24,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:24,486 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,486 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:24,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:24,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:24,486 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:24,486 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:24,486 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,486 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,486 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,486 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:24,487 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:24,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:24,587 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:24,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:24,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:24,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:24,596 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/0b591b582287486cb9f222ebad5bcc1f 2023-04-13 14:55:24,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:24,596 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:24,596 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:24,596 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:24,596 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:24,597 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,597 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,598 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,598 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:24,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:24,598 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,598 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:24,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:24,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:24,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/3ecfed8f6ffe48c9a1ef0ac8507b4a41 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/3ecfed8f6ffe48c9a1ef0ac8507b4a41 2023-04-13 14:55:24,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/3ecfed8f6ffe48c9a1ef0ac8507b4a41, entries=200, sequenceid=213, filesize=32.1 K 2023-04-13 14:55:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:24,606 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,606 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:24,606 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:24,606 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:24,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/86536308cd6541f295cadd068b196f32 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/86536308cd6541f295cadd068b196f32 2023-04-13 14:55:24,608 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,608 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,609 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,609 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:24,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a3f4cb7[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:24,610 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:24,610 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,610 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/86536308cd6541f295cadd068b196f32, entries=150, sequenceid=213, filesize=11.8 K 2023-04-13 14:55:24,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/0b591b582287486cb9f222ebad5bcc1f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/0b591b582287486cb9f222ebad5bcc1f 2023-04-13 14:55:24,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/0b591b582287486cb9f222ebad5bcc1f, entries=150, sequenceid=213, filesize=11.8 K 2023-04-13 14:55:24,615 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7763c91103ebbe310342746ed8aa8aa1 in 1270ms, sequenceid=213, compaction requested=true 2023-04-13 14:55:24,615 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:24,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:24,615 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 14:55:24,615 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 14:55:24,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:24,616 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:24,616 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,617 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:24,617 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,617 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:24,617 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:24,617 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:24,619 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,619 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,619 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:24,620 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:24,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:24,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:24,622 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:24,622 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:24,622 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:24,622 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:24,622 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:24,623 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,623 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,623 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:24,624 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,624 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57876 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 14:55:24,624 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7763c91103ebbe310342746ed8aa8aa1/B is initiating minor compaction (all files) 2023-04-13 14:55:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,625 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7763c91103ebbe310342746ed8aa8aa1/B in TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,625 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/134e4d6db0ae45a88640a3b2724984a2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/48dc3c99f77d4b488dcf26bee09c01fc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/5189738cd7844f0299ac2cdf71c019d6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/de3f25d68f73496d8208d8e796112589, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/86536308cd6541f295cadd068b196f32] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp, totalSize=56.5 K 2023-04-13 14:55:24,625 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136552 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 14:55:24,625 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7763c91103ebbe310342746ed8aa8aa1/A is initiating minor compaction (all files) 2023-04-13 14:55:24,625 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7763c91103ebbe310342746ed8aa8aa1/A in TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,625 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/cd235594f022426fb740bac411811466, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/a27b06e47b34446ba704957043008c9e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/28791ce94b4b4d68bd29b06bf734d01d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/ca8477f869da4aabbcd0f970ce834111, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/3ecfed8f6ffe48c9a1ef0ac8507b4a41] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp, totalSize=133.4 K 2023-04-13 14:55:24,625 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 134e4d6db0ae45a88640a3b2724984a2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681397718355 2023-04-13 14:55:24,626 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting cd235594f022426fb740bac411811466, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681397718355 2023-04-13 14:55:24,626 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 48dc3c99f77d4b488dcf26bee09c01fc, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681397719485 2023-04-13 14:55:24,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,626 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting a27b06e47b34446ba704957043008c9e, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681397719485 2023-04-13 14:55:24,627 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 5189738cd7844f0299ac2cdf71c019d6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681397721679 2023-04-13 14:55:24,627 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 28791ce94b4b4d68bd29b06bf734d01d, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681397721679 2023-04-13 14:55:24,627 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting de3f25d68f73496d8208d8e796112589, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681397722059 2023-04-13 14:55:24,627 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting ca8477f869da4aabbcd0f970ce834111, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681397722059 2023-04-13 14:55:24,628 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 86536308cd6541f295cadd068b196f32, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681397723230 2023-04-13 14:55:24,628 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 3ecfed8f6ffe48c9a1ef0ac8507b4a41, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681397723229 2023-04-13 14:55:24,632 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,635 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:24,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:24,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:24,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397723469, max timestamp=1681397724634Num uniques 1018; 2023-04-13 14:55:24,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 14:55:24,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397723469, max timestamp=1681397724634Num uniques 1018; 2023-04-13 14:55:24,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 14:55:24,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397723469, max timestamp=1681397724634Num uniques 1018; 2023-04-13 14:55:24,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 14:55:24,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:24,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:24,638 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:24,638 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7763c91103ebbe310342746ed8aa8aa1#B#compaction#335 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:24,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,638 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:24,638 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:24,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:24,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:24,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:24,650 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7763c91103ebbe310342746ed8aa8aa1#A#compaction#336 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:24,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:24,655 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=53.67 KB heapSize=145.08 KB 2023-04-13 14:55:24,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:24,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:24,656 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:24,656 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:24,656 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:24,656 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:24,661 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/6c0aaccc09474165811d7b1f5811b658 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/6c0aaccc09474165811d7b1f5811b658 2023-04-13 14:55:24,667 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7763c91103ebbe310342746ed8aa8aa1/B of 7763c91103ebbe310342746ed8aa8aa1 into 6c0aaccc09474165811d7b1f5811b658(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:24,667 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:24,667 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., storeName=7763c91103ebbe310342746ed8aa8aa1/B, priority=11, startTime=1681397724615; duration=0sec 2023-04-13 14:55:24,667 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:24,667 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 14:55:24,674 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57876 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 14:55:24,674 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7763c91103ebbe310342746ed8aa8aa1/C is initiating minor compaction (all files) 2023-04-13 14:55:24,674 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7763c91103ebbe310342746ed8aa8aa1/C in TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,674 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/8f603c721fdb402a85c49ff4bc3239fc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/abca699ce6c74e63b02c11b7b0f890e5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/430e970922da4c659e0094096c1e5ccd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/ff03f30cd5794974b9982ccb9c767562, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/0b591b582287486cb9f222ebad5bcc1f] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp, totalSize=56.5 K 2023-04-13 14:55:24,675 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 8f603c721fdb402a85c49ff4bc3239fc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681397718355 2023-04-13 14:55:24,677 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting abca699ce6c74e63b02c11b7b0f890e5, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681397719485 2023-04-13 14:55:24,677 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 430e970922da4c659e0094096c1e5ccd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681397721679 2023-04-13 14:55:24,678 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting ff03f30cd5794974b9982ccb9c767562, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681397722059 2023-04-13 14:55:24,679 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 0b591b582287486cb9f222ebad5bcc1f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681397723230 2023-04-13 14:55:24,733 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7763c91103ebbe310342746ed8aa8aa1#C#compaction#338 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:24,738 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:24,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:24,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:24,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:24,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:24,748 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:24,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:24,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:24,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:24,749 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,749 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,751 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:24,751 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:24,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:24,751 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/4a63b8a39b3b426e97adbe9b0d752640 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4a63b8a39b3b426e97adbe9b0d752640 2023-04-13 14:55:24,751 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,751 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:24,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:24,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:24,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:24,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:24,752 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:24,752 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:24,752 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:24,754 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:24,754 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:24,754 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,754 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:24,754 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:24,754 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,754 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:24,754 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:24,754 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,754 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:24,754 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:24,755 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,755 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,756 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,756 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,756 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7763c91103ebbe310342746ed8aa8aa1/C of 7763c91103ebbe310342746ed8aa8aa1 into 4a63b8a39b3b426e97adbe9b0d752640(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,756 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:24,756 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., storeName=7763c91103ebbe310342746ed8aa8aa1/C, priority=11, startTime=1681397724615; duration=0sec 2023-04-13 14:55:24,757 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:24,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:24,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24a0b76[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:24,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:24,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,759 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:24,759 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:24,759 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,759 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:24,759 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:24,760 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,760 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:24,760 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:24,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,760 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:24,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:24,762 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:24,762 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:24,762 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:24,762 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:24,762 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:24,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,766 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,769 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:24,769 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:24,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:24,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:24,770 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,770 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,770 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,770 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,770 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:24,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:24,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:24,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:24,770 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:24,770 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,770 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,770 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:24,770 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:24,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:24,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397724766, max timestamp=1681397724787Num uniques 859; 2023-04-13 14:55:24,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:24,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397724766, max timestamp=1681397724787Num uniques 859; 2023-04-13 14:55:24,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397724766, max timestamp=1681397724787Num uniques 859; 2023-04-13 14:55:24,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:24,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:24,870 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:24,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:24,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:24,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:24,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:24,874 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:24,874 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:24,874 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:24,874 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:24,874 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,874 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,875 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,875 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:24,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:24,875 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,875 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:24,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:24,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:24,880 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:24,880 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:24,880 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:24,880 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:24,880 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:24,880 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,880 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:24,880 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:24,881 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:24,881 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:24,881 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:24,881 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,881 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:24,881 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:24,882 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,882 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,882 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,882 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:24,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:24,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d72a6b4[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:24,883 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:24,883 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,886 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,886 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:24,886 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:24,886 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,886 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:24,886 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:24,887 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,887 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,887 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:24,887 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:24,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:24,889 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:24,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:24,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:24,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:24,889 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:24,890 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,890 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:24,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,890 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:24,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:24,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:24,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:24,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:24,891 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,892 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:24,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,895 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:24,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:24,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:24,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:24,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,896 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:24,896 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:24,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:24,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:24,896 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:24,896 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:24,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:24,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:24,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:24,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:24,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397784891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:24,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397784892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:24,996 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:24,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:24,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:24,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:24,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:24,999 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:24,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:24,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:24,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:24,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:24,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,001 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:25,001 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,001 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,001 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:25,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:25,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:25,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:25,002 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:25,002 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:25,002 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:25,002 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:25,002 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:25,002 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:25,002 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:25,002 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:25,003 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:25,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:25,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:25,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:25,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:25,004 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,004 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:25,004 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,004 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:25,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:25,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f5d29bf[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:25,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:25,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,007 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,008 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:25,008 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:25,008 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,008 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:25,008 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:25,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:25,010 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:25,010 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:25,010 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,010 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:25,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397785006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:25,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:25,013 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:25,013 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:25,013 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:25,013 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:25,013 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:25,014 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,014 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:25,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:25,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,014 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:25,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:25,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:25,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397785006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:25,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:25,015 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:25,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,016 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:25,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,018 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:25,018 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:25,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:25,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:25,018 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:25,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:25,018 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:25,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:25,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:25,018 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,019 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:25,019 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,019 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:25,019 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,019 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,019 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,019 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,119 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:25,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:25,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:25,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:25,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:25,122 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:25,122 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:25,122 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:25,122 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:25,122 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,122 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:25,123 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:25,124 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,124 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:25,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:25,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:25,125 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:25,125 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:25,125 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:25,125 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:25,125 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:25,125 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:25,126 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:25,126 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:25,126 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:25,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:25,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:25,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:25,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:25,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,130 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:25,130 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:25,131 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304134a2a3d8f10be47f09f2224e950d10134 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134a2a3d8f10be47f09f2224e950d10134 2023-04-13 14:55:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,137 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=255, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/160c7a7958b340dd899832f2bcba02eb 2023-04-13 14:55:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:25,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:25,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f078617[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:25,140 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:25,140 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,141 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,141 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:25,141 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:25,141 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:25,141 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,142 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:25,143 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:25,143 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,143 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:25,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:25,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:25,143 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:25,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:25,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:25,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:25,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:25,145 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:25,145 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:25,145 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:25,146 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:25,146 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:25,147 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,147 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:25,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,147 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:25,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:25,148 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:25,148 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/078c27598f7d49e6bf5758037c3057db 2023-04-13 14:55:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,149 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,154 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,154 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,154 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:25,154 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:25,154 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,154 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:25,154 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:25,154 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:25,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:25,154 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,154 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:25,155 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:25,155 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:25,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:25,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397785213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:25,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397785216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,255 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:25,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:25,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:25,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:25,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:25,258 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:25,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:25,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:25,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:25,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,259 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:25,259 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:25,260 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,260 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:25,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:25,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:25,261 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:25,261 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:25,261 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:25,261 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:25,263 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,263 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:25,263 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,263 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:25,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:25,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d442d3a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:25,264 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:25,264 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,265 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:25,265 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:25,265 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:25,266 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,266 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:25,267 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,268 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:25,268 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:25,268 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:25,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:25,270 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:25,270 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:25,270 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:25,270 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:25,270 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:25,271 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,271 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:25,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:25,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,271 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:25,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:25,272 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,273 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,275 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,275 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,275 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,275 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,275 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:25,275 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:25,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:25,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:25,275 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,275 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,275 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:25,275 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:25,275 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:25,275 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:25,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:25,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:25,376 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:25,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:25,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:25,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:25,378 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:25,379 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:25,379 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:25,379 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:25,379 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,379 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,384 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:25,384 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:25,384 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,384 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:25,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:25,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:25,386 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:25,386 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:25,386 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:25,386 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:25,386 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:25,386 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:25,386 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:25,386 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:25,386 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:25,386 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:25,386 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:25,386 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,386 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:25,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:25,388 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,388 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:25,388 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,388 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:25,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:25,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15cb6720[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:25,390 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:25,390 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,391 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,391 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:25,396 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:25,396 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:25,396 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:25,397 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,397 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:25,397 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:25,397 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:25,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:25,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:25,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:25,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:25,399 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:25,400 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:25,400 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:25,400 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:25,400 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:25,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:25,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:25,403 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,405 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,407 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,407 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,407 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,407 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,407 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,407 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:25,407 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:25,407 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:25,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:25,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:25,407 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,407 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:25,407 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:25,408 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:25,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:25,508 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:25,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:25,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:25,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:25,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:25,511 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:25,511 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:25,511 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:25,511 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:25,511 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,511 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,512 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:25,512 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:25,513 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,513 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:25,514 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:25,514 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:25,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:25,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:25,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:25,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:25,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:25,514 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:25,514 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:25,514 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:25,514 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:25,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:25,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:25,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:25,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:25,515 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:25,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,516 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:25,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:25,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@733db4f7[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:25,517 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:25,518 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,523 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,523 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:25,523 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:25,523 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:25,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:25,525 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:25,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:25,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:25,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:25,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:25,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:25,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:25,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:25,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:25,527 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:25,527 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:25,527 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:25,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:25,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:25,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397785521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,528 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:25,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:25,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397785524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,529 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,529 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:25,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:25,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,529 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:25,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:25,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:25,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:25,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,530 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,532 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:25,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:25,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:25,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:25,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:25,537 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:25,537 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:25,537 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,538 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,538 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,538 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,538 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,538 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:25,538 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:25,537 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:25,540 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:25,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:25,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:25,564 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/48f89914110445edacfa4b63188f199c 2023-04-13 14:55:25,568 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/160c7a7958b340dd899832f2bcba02eb as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/160c7a7958b340dd899832f2bcba02eb 2023-04-13 14:55:25,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/160c7a7958b340dd899832f2bcba02eb, entries=200, sequenceid=255, filesize=32.1 K 2023-04-13 14:55:25,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/078c27598f7d49e6bf5758037c3057db as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/078c27598f7d49e6bf5758037c3057db 2023-04-13 14:55:25,578 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/078c27598f7d49e6bf5758037c3057db, entries=150, sequenceid=255, filesize=11.8 K 2023-04-13 14:55:25,579 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/48f89914110445edacfa4b63188f199c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/48f89914110445edacfa4b63188f199c 2023-04-13 14:55:25,583 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/48f89914110445edacfa4b63188f199c, entries=150, sequenceid=255, filesize=11.8 K 2023-04-13 14:55:25,583 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.03 KB/148512, currentSize=136.42 KB/139690 for 7763c91103ebbe310342746ed8aa8aa1 in 928ms, sequenceid=255, compaction requested=false 2023-04-13 14:55:25,584 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:25,637 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:25,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:25,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:25,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:25,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:25,644 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:25,644 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:25,644 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:25,644 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:25,644 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,644 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,648 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:25,648 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:25,649 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:25,649 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:25,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:25,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:25,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:25,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:25,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:25,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:25,651 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:25,651 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:25,651 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:25,651 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:25,651 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:25,651 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:25,651 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:25,651 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=136.42 KB heapSize=390.06 KB 2023-04-13 14:55:25,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:25,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:25,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:25,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:25,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:25,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:25,940 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/d1005df7cf754718b7eff189653fe52c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/d1005df7cf754718b7eff189653fe52c 2023-04-13 14:55:25,944 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7763c91103ebbe310342746ed8aa8aa1/A of 7763c91103ebbe310342746ed8aa8aa1 into d1005df7cf754718b7eff189653fe52c(size=25.7 K), total size for store is 57.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-13 14:55:25,944 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:25,944 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., storeName=7763c91103ebbe310342746ed8aa8aa1/A, priority=11, startTime=1681397724615; duration=1sec 2023-04-13 14:55:25,944 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:26,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:26,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:26,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:26,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397786055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:26,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397786056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,076 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304136e320c685b874873aeaea9fcc8aa10f2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136e320c685b874873aeaea9fcc8aa10f2 2023-04-13 14:55:26,081 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=294, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/0b67dac80ea2401bbf21aee7edf29f93 2023-04-13 14:55:26,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:26,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397786162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:26,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397786162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:26,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397786369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:26,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397786370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,490 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/26a64a9e7fec460c9092e37d3f5bcbc6 2023-04-13 14:55:26,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:26,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57326 deadline: 1681397786542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,550 DEBUG [Thread-2236] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8176 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:55:26,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:26,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57290 deadline: 1681397786562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,568 DEBUG [Thread-2242] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8192 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:55:26,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:26,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57288 deadline: 1681397786568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,575 DEBUG [Thread-2238] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8194 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:55:26,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:26,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397786675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:26,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397786675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,900 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/4b7a2a600243450281fe8948b7fb2def 2023-04-13 14:55:26,904 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/0b67dac80ea2401bbf21aee7edf29f93 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/0b67dac80ea2401bbf21aee7edf29f93 2023-04-13 14:55:26,907 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/0b67dac80ea2401bbf21aee7edf29f93, entries=150, sequenceid=294, filesize=25.4 K 2023-04-13 14:55:26,908 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-04-13 14:55:26,908 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/26a64a9e7fec460c9092e37d3f5bcbc6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/26a64a9e7fec460c9092e37d3f5bcbc6 2023-04-13 14:55:26,912 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/26a64a9e7fec460c9092e37d3f5bcbc6, entries=150, sequenceid=294, filesize=11.9 K 2023-04-13 14:55:26,913 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/4b7a2a600243450281fe8948b7fb2def as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4b7a2a600243450281fe8948b7fb2def 2023-04-13 14:55:26,917 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4b7a2a600243450281fe8948b7fb2def, entries=150, sequenceid=294, filesize=11.9 K 2023-04-13 14:55:26,918 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~390.02 KB/399376, currentSize=46.96 KB/48090 for 7763c91103ebbe310342746ed8aa8aa1 in 1267ms, sequenceid=294, compaction requested=true 2023-04-13 14:55:26,918 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:26,918 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:26,918 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:26,918 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:26,918 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,918 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:26,918 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:26,919 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:26,919 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:26,920 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:26,920 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:26,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:26,921 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:26,921 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:26,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4039b3b9[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:26,922 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:26,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:26,923 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:26,923 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,923 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:26,923 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:26,923 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:26,924 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:26,924 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:26,924 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,924 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:26,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:26,927 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:26,927 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:26,927 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:26,927 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:26,927 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:26,931 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:26,932 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:26,932 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:26,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:26,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:26,932 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:26,933 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,936 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:26,936 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:26,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:26,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:26,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:26,936 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,936 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:26,936 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:26,936 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:26,936 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:26,936 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:26,936 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:26,936 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:26,936 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:26,936 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:26,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:26,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:27,036 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:27,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:27,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:27,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:27,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:27,041 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:27,041 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:27,041 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:27,041 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:27,041 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:27,041 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:27,042 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:27,042 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:27,043 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,043 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:27,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:27,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:27,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:27,049 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:27,049 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:27,049 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:55:27,049 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:27,049 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:27,049 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:27,049 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:27,049 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:27,049 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:27,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:27,195 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:27,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397727243, max timestamp=1681397727258Num uniques 859; 2023-04-13 14:55:27,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:27,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397727243, max timestamp=1681397727258Num uniques 859; 2023-04-13 14:55:27,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:27,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397727243, max timestamp=1681397727258Num uniques 859; 2023-04-13 14:55:27,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:27,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:27,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397787315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:27,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397787317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:27,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397787423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:27,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397787425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,486 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304137954c51b5f2c47059807f316c1405e5d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137954c51b5f2c47059807f316c1405e5d 2023-04-13 14:55:27,490 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=305, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/b50c0112c9a643df8a94802736afa556 2023-04-13 14:55:27,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:27,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397787630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:27,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397787630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,899 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/af802aec94bd40c7b8de59443907c8bb 2023-04-13 14:55:27,931 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/b30fdf92042b4559a7e65d6433e879a4 2023-04-13 14:55:27,937 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/b50c0112c9a643df8a94802736afa556 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/b50c0112c9a643df8a94802736afa556 2023-04-13 14:55:27,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:27,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397787937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:27,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397787937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,942 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/b50c0112c9a643df8a94802736afa556, entries=100, sequenceid=305, filesize=18.6 K 2023-04-13 14:55:27,943 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/af802aec94bd40c7b8de59443907c8bb as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/af802aec94bd40c7b8de59443907c8bb 2023-04-13 14:55:27,949 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/af802aec94bd40c7b8de59443907c8bb, entries=100, sequenceid=305, filesize=9.5 K 2023-04-13 14:55:27,950 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/b30fdf92042b4559a7e65d6433e879a4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b30fdf92042b4559a7e65d6433e879a4 2023-04-13 14:55:27,954 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b30fdf92042b4559a7e65d6433e879a4, entries=100, sequenceid=305, filesize=9.5 K 2023-04-13 14:55:27,955 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7763c91103ebbe310342746ed8aa8aa1 in 906ms, sequenceid=305, compaction requested=true 2023-04-13 14:55:27,955 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:27,955 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:27,955 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:27,955 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:27,955 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,955 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:27,955 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:27,960 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:27,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:27,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:27,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:27,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:27,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c686b79[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:27,964 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:27,964 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:27,965 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:27,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:27,965 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:27,965 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:27,965 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:27,965 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,966 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:27,967 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:27,967 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,967 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:27,967 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:27,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:27,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:27,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:27,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:27,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:27,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:27,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:27,970 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:27,970 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:27,970 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:27,970 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:27,970 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:27,972 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:27,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:27,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:27,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:27,973 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:27,973 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:27,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:27,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:27,975 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:27,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:27,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:27,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:27,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:27,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,976 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:27,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:27,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:27,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:27,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:27,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:27,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:27,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:27,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:27,979 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:27,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:27,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:27,979 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:27,980 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:27,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:27,980 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:27,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:27,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:27,980 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:28,080 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:28,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:28,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:28,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:28,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:28,083 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:28,083 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:28,084 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:28,084 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:28,084 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:28,084 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:28,085 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:28,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:28,085 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:28,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:28,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:28,087 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:28,087 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:28,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:28,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:28,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:28,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:28,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:28,092 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:28,092 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:28,092 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:28,092 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:28,092 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-13 14:55:28,093 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:28,093 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:28,093 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:28,093 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:28,093 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:28,093 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:28,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:28,446 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:28,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:28,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397788461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:28,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:28,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397788469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:28,521 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041301dc3fcfb46247d1ab1b5afd548e435a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041301dc3fcfb46247d1ab1b5afd548e435a 2023-04-13 14:55:28,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:28,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397788569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:28,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:28,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397788573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:28,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:28,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397788782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:28,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:28,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397788784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:28,927 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=342, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/976203fd1e5a4935a4e00de717895738 2023-04-13 14:55:28,977 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/affacc8e618548dca01ca135816d99cd 2023-04-13 14:55:29,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:29,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397789090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:29,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397789095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,391 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/a17bfb3c9f6b49529f76dcba50723c4f 2023-04-13 14:55:29,396 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/976203fd1e5a4935a4e00de717895738 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/976203fd1e5a4935a4e00de717895738 2023-04-13 14:55:29,401 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/976203fd1e5a4935a4e00de717895738, entries=150, sequenceid=342, filesize=25.4 K 2023-04-13 14:55:29,402 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/affacc8e618548dca01ca135816d99cd as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/affacc8e618548dca01ca135816d99cd 2023-04-13 14:55:29,405 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/affacc8e618548dca01ca135816d99cd, entries=150, sequenceid=342, filesize=11.9 K 2023-04-13 14:55:29,406 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/a17bfb3c9f6b49529f76dcba50723c4f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/a17bfb3c9f6b49529f76dcba50723c4f 2023-04-13 14:55:29,410 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/a17bfb3c9f6b49529f76dcba50723c4f, entries=150, sequenceid=342, filesize=11.9 K 2023-04-13 14:55:29,410 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 7763c91103ebbe310342746ed8aa8aa1 in 1318ms, sequenceid=342, compaction requested=true 2023-04-13 14:55:29,410 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:29,410 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:29,410 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:29,410 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:29,411 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,411 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:29,411 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:29,412 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:29,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:29,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:29,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:29,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:29,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:29,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:29,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7254f9b3[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:29,414 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:29,414 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:29,415 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:29,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:29,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:29,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:29,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:29,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:29,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:29,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:29,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:29,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:29,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:29,418 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:29,419 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:29,419 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:29,419 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:29,419 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:29,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:29,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:29,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:29,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:29,422 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:29,422 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,424 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:29,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:29,426 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,426 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:29,426 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:29,426 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:29,426 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:29,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:29,426 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:29,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:29,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:29,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:29,426 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:29,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:29,426 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:29,426 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:29,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:29,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:29,526 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:29,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:29,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:29,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:29,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:29,535 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:29,536 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:29,536 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:29,536 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:29,536 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:29,536 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:29,537 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:29,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:29,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:29,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:29,538 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,538 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:29,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:29,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:29,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:29,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:29,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:29,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:29,539 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:29,539 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:29,539 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:29,539 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:29,539 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:29,539 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:29,539 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:29,539 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:55:29,539 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:29,540 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:29,540 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:29,540 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:29,540 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:29,540 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:29,575 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413fb41bdf5c4984eeeb571508d528c3d50 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413fb41bdf5c4984eeeb571508d528c3d50 2023-04-13 14:55:29,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:29,602 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:29,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397729626, max timestamp=1681397729693Num uniques 859; 2023-04-13 14:55:29,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:29,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397729626, max timestamp=1681397729693Num uniques 859; 2023-04-13 14:55:29,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397729626, max timestamp=1681397729693Num uniques 859; 2023-04-13 14:55:29,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:29,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:29,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:29,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397789768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:29,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397789771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:29,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397789878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:29,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397789883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:29,994 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=352, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/0204162e49434709a49d948d26244ffa 2023-04-13 14:55:30,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:30,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397790086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:30,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397790090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:30,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397790396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,404 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/ff9da4465414482582e0c2bd6e9ac3f1 2023-04-13 14:55:30,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:30,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397790398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,814 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/8557edaf32b54516a42ac91a27c91d32 2023-04-13 14:55:30,818 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/0204162e49434709a49d948d26244ffa as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/0204162e49434709a49d948d26244ffa 2023-04-13 14:55:30,822 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/0204162e49434709a49d948d26244ffa, entries=150, sequenceid=352, filesize=25.4 K 2023-04-13 14:55:30,823 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/ff9da4465414482582e0c2bd6e9ac3f1 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/ff9da4465414482582e0c2bd6e9ac3f1 2023-04-13 14:55:30,826 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/ff9da4465414482582e0c2bd6e9ac3f1, entries=150, sequenceid=352, filesize=11.9 K 2023-04-13 14:55:30,827 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/8557edaf32b54516a42ac91a27c91d32 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/8557edaf32b54516a42ac91a27c91d32 2023-04-13 14:55:30,831 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/8557edaf32b54516a42ac91a27c91d32, entries=150, sequenceid=352, filesize=11.9 K 2023-04-13 14:55:30,832 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7763c91103ebbe310342746ed8aa8aa1 in 1293ms, sequenceid=352, compaction requested=true 2023-04-13 14:55:30,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:30,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:30,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:30,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:30,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,833 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:30,833 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:30,834 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:30,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:30,842 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:30,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c507102[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:30,842 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,843 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,843 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:30,843 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:30,843 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:30,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:30,846 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:30,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:30,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:30,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:30,848 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:30,848 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:30,848 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:30,848 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:30,848 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:30,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:30,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,853 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:30,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:30,854 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:30,858 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:30,858 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:30,859 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:30,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:30,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:30,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:30,859 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:30,859 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:30,859 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,859 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,859 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,859 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:30,859 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:30,859 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:30,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:30,859 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:30,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:30,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:30,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-13 14:55:30,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:30,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:30,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:30,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:30,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:30,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:30,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:30,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397790930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:30,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397790934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,959 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:30,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:30,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:30,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:30,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:30,963 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:30,963 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:30,963 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:30,963 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:30,963 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,963 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:30,965 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:30,965 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:30,965 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,965 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:30,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:30,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:30,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:30,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:30,967 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:30,967 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:30,967 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:30,967 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,968 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,970 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,970 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:30,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:30,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6179a34a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:30,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:30,973 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,974 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:30,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:30,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:30,976 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,976 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:30,980 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:30,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:30,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:30,980 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:30,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:30,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:30,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:30,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:30,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:30,982 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:30,982 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:30,982 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:30,982 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:30,982 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:30,984 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,984 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,984 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:30,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:30,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:30,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:30,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:30,984 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:30,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:30,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:30,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:30,985 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:30,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:30,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,989 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,989 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:30,989 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:30,989 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:30,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:30,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:30,989 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:30,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:30,989 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:30,990 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:30,990 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:30,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:30,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:30,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:30,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:30,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:31,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397791037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:31,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397791046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,090 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:31,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:31,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:31,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:31,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:31,093 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:31,093 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,093 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:31,093 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:31,094 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,094 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,095 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,095 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,095 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,095 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:31,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:31,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:31,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:31,097 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:31,097 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,097 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:31,097 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:31,097 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:31,097 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,097 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:31,097 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:31,098 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,098 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:31,098 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:31,098 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,098 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:31,098 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:31,100 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:31,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e352fbf[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:31,102 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:31,102 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,103 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,104 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:31,104 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,104 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:31,104 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:31,104 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:31,105 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:31,105 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,105 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,105 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:31,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:31,107 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:31,107 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:31,107 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:31,108 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:31,108 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:31,109 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,109 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,109 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,110 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,111 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:31,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:31,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:31,114 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,114 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:31,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,114 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:31,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,215 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:31,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:31,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:31,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:31,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:31,218 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:31,218 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,218 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:31,218 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:31,218 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,218 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,219 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,219 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,220 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,220 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:31,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:31,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:31,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:31,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:31,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:31,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:31,221 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,221 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:31,221 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:31,221 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:31,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:31,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:31,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:31,223 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,223 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,223 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,223 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:31,224 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:31,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bf9c4ab[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:31,224 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,225 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,225 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:31,225 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:31,225 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,225 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:31,225 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:31,227 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,227 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,227 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,227 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:31,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:31,230 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:31,230 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:31,230 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:31,230 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:31,230 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:31,231 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,231 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,231 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,231 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,233 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,234 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,234 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,234 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,234 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:31,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:31,234 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,234 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,235 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:31,235 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:31,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,235 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,235 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,235 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,235 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:31,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:31,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397791247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:31,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397791254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,323 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413d873f13c283d4022b937748fb4140696 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d873f13c283d4022b937748fb4140696 2023-04-13 14:55:31,328 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=388, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/d9689f44ee8646788bf22046c569c32a 2023-04-13 14:55:31,335 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:31,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:31,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:31,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:31,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:31,341 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:31,341 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,341 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:31,341 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:31,341 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,341 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,342 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,342 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,343 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,343 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:31,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:31,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:31,348 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:31,348 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:31,348 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,348 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:31,348 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:31,348 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,348 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:31,348 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:31,348 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,348 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:31,348 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:31,350 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,351 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,351 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,351 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:31,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24ce64b7[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:31,352 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:31,353 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,354 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,354 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:31,354 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:31,354 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:31,354 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,354 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:31,355 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:31,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:31,357 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:31,357 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:31,357 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:31,357 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:31,357 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:31,363 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,363 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,363 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,363 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,365 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,366 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,367 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,367 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,367 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,367 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:31,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:31,367 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,367 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,367 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,367 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,367 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,367 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:31,367 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:31,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,467 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:31,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:31,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:31,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:31,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:31,470 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:31,470 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,470 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:31,470 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:31,471 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,471 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,473 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,473 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,474 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,474 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:31,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:31,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:31,475 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:31,475 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,475 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:31,475 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:31,475 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:31,475 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,475 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:31,475 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:31,475 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,475 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:31,475 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:31,475 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,476 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:31,476 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:31,477 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,477 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,477 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,477 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:31,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@881f097[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:31,479 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:31,479 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,480 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,480 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:31,480 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:31,480 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:31,480 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,480 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:31,481 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,481 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:31,481 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,481 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:31,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:31,483 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:31,483 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:31,483 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:31,483 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:31,483 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:31,485 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,485 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,485 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,486 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,487 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,489 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,489 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,489 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:31,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:31,489 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,489 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,489 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:31,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:31,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,489 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,489 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:31,490 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,490 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,490 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,490 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:31,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397791557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:31,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397791563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,590 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:31,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:31,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:31,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:31,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:31,593 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:31,593 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,593 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:31,593 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:31,593 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,593 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,594 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,595 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,595 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:31,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:31,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:31,595 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:31,595 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:31,596 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,596 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:31,596 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:31,596 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:31,599 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,599 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,600 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,600 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:31,601 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:31,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ffe6eef[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:31,601 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,602 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,602 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:31,602 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:31,602 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:31,602 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,602 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:31,604 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,604 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:31,604 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,604 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:31,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:31,607 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:31,607 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:31,607 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:31,607 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:31,607 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:31,608 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,609 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,609 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,610 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:31,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:31,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:31,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,612 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,613 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:31,613 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:31,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,713 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:31,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:31,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:31,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:31,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:31,716 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:31,716 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,716 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:31,716 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:31,716 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,716 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,718 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,718 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,718 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,718 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:31,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:31,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:31,719 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:31,719 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,719 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:31,719 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:31,719 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:31,719 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,719 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:31,719 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:31,719 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:31,722 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,722 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,722 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,722 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:31,724 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:31,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a272927[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:31,724 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,725 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,725 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,725 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:31,725 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:31,725 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:31,725 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:31,726 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,726 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,726 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,726 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:31,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:31,729 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:31,729 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:31,729 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:31,729 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:31,729 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:31,730 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,730 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,730 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,731 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,732 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,738 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/1f5ff4eb11684f9e9b28780ff7707510 2023-04-13 14:55:31,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,739 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:31,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:31,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:31,740 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:31,740 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:31,840 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:31,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:31,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:31,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:31,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:31,843 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:31,843 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,843 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:31,843 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:31,843 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,845 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:31,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:31,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:31,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:31,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:31,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:31,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:31,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:31,846 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:31,846 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,846 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:31,846 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:31,846 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:31,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:31,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:31,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:31,848 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,848 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,848 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,848 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:31,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:31,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d9b1908[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:31,850 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:31,850 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,851 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:31,851 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,851 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:31,851 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:31,851 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:31,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,853 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:31,853 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,853 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:31,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:31,856 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:31,856 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:31,856 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:31,856 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:31,856 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:31,857 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,857 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,857 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,858 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,859 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,861 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,861 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:31,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:31,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:31,861 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,861 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,861 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,862 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:31,862 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:31,862 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,862 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,862 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,862 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,862 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,962 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:31,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:31,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:31,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:31,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:31,965 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:31,965 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,965 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:31,965 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:31,965 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,966 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,967 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,967 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,967 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,967 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:31,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:31,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:31,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:31,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:31,968 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:31,968 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:31,968 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:31,968 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:31,968 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:31,969 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,969 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:31,969 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:31,969 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:31,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:31,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:31,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:31,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:31,971 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,971 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,971 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,971 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:31,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:31,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a995964[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:31,973 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:31,973 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,974 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:31,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:31,974 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,974 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:31,974 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:31,975 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,975 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,975 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,975 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:31,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:31,977 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:31,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:31,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:31,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:31,977 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:31,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,980 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:31,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,980 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:31,980 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:31,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:31,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,981 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:31,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:31,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,988 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:31,988 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:31,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:31,988 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,988 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:31,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:31,988 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:31,988 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:31,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:31,988 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:31,988 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:31,988 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:31,989 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,989 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:31,989 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:32,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:32,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397792068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:32,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397792069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,089 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:32,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:32,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:32,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:32,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:32,096 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:32,096 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:32,097 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:32,097 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:32,097 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:32,097 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:32,098 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,098 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:32,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:32,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:32,098 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,098 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:32,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:32,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:32,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:32,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:32,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:32,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:32,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:32,099 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:32,099 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:32,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:32,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:32,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:32,100 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:32,100 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:32,100 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:32,100 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:32,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:32,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:32,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:32,100 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:32,101 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,101 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:32,102 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:32,102 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:32,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:32,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:32,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26dccbce[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:32,103 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:32,103 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:32,104 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:32,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:32,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:32,104 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:32,104 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:32,104 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:32,104 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,104 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:32,106 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:32,106 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:32,106 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,106 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:32,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:32,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:32,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:32,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:32,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:32,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:32,109 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:32,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:32,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:32,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:32,109 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:32,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:32,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:32,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:32,111 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:32,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:32,111 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:32,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:32,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:32,112 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:32,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:32,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:32,117 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:32,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:32,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:32,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:32,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:32,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:32,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:32,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:32,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:32,117 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:32,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:32,117 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:32,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:32,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:32,152 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/e7151b18e6754ff0be65b7d4201e93c5 2023-04-13 14:55:32,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/d9689f44ee8646788bf22046c569c32a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/d9689f44ee8646788bf22046c569c32a 2023-04-13 14:55:32,159 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/d9689f44ee8646788bf22046c569c32a, entries=200, sequenceid=388, filesize=32.3 K 2023-04-13 14:55:32,159 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/1f5ff4eb11684f9e9b28780ff7707510 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/1f5ff4eb11684f9e9b28780ff7707510 2023-04-13 14:55:32,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/1f5ff4eb11684f9e9b28780ff7707510, entries=150, sequenceid=388, filesize=11.9 K 2023-04-13 14:55:32,163 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/e7151b18e6754ff0be65b7d4201e93c5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/e7151b18e6754ff0be65b7d4201e93c5 2023-04-13 14:55:32,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/e7151b18e6754ff0be65b7d4201e93c5, entries=150, sequenceid=388, filesize=11.9 K 2023-04-13 14:55:32,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 7763c91103ebbe310342746ed8aa8aa1 in 1268ms, sequenceid=388, compaction requested=true 2023-04-13 14:55:32,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:32,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:32,174 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 14:55:32,174 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 14:55:32,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:32,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:32,176 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 189414 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 14:55:32,176 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7763c91103ebbe310342746ed8aa8aa1/A is initiating minor compaction (all files) 2023-04-13 14:55:32,176 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7763c91103ebbe310342746ed8aa8aa1/A in TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:32,176 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/d1005df7cf754718b7eff189653fe52c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/160c7a7958b340dd899832f2bcba02eb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/0b67dac80ea2401bbf21aee7edf29f93, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/b50c0112c9a643df8a94802736afa556, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/976203fd1e5a4935a4e00de717895738, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/0204162e49434709a49d948d26244ffa, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/d9689f44ee8646788bf22046c569c32a] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp, totalSize=185.0 K 2023-04-13 14:55:32,176 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting d1005df7cf754718b7eff189653fe52c, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681397723230 2023-04-13 14:55:32,176 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 82968 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 14:55:32,177 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7763c91103ebbe310342746ed8aa8aa1/B is initiating minor compaction (all files) 2023-04-13 14:55:32,177 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7763c91103ebbe310342746ed8aa8aa1/B in TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:32,177 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/6c0aaccc09474165811d7b1f5811b658, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/078c27598f7d49e6bf5758037c3057db, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/26a64a9e7fec460c9092e37d3f5bcbc6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/af802aec94bd40c7b8de59443907c8bb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/affacc8e618548dca01ca135816d99cd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/ff9da4465414482582e0c2bd6e9ac3f1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/1f5ff4eb11684f9e9b28780ff7707510] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp, totalSize=81.0 K 2023-04-13 14:55:32,177 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 160c7a7958b340dd899832f2bcba02eb, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1681397723477 2023-04-13 14:55:32,177 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 6c0aaccc09474165811d7b1f5811b658, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681397723230 2023-04-13 14:55:32,177 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 0b67dac80ea2401bbf21aee7edf29f93, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681397724853 2023-04-13 14:55:32,178 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 078c27598f7d49e6bf5758037c3057db, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1681397723477 2023-04-13 14:55:32,179 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting b50c0112c9a643df8a94802736afa556, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1681397726054 2023-04-13 14:55:32,179 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 26a64a9e7fec460c9092e37d3f5bcbc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681397724853 2023-04-13 14:55:32,179 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 976203fd1e5a4935a4e00de717895738, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681397727307 2023-04-13 14:55:32,179 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting af802aec94bd40c7b8de59443907c8bb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1681397726054 2023-04-13 14:55:32,179 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 0204162e49434709a49d948d26244ffa, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1681397728451 2023-04-13 14:55:32,179 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting affacc8e618548dca01ca135816d99cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681397727307 2023-04-13 14:55:32,180 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting d9689f44ee8646788bf22046c569c32a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1681397729740 2023-04-13 14:55:32,180 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting ff9da4465414482582e0c2bd6e9ac3f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1681397728451 2023-04-13 14:55:32,180 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 1f5ff4eb11684f9e9b28780ff7707510, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1681397729740 2023-04-13 14:55:32,191 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7763c91103ebbe310342746ed8aa8aa1#B#compaction#357 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:32,199 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7763c91103ebbe310342746ed8aa8aa1#A#compaction#356 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:32,218 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:32,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:32,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:32,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:32,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:32,223 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:32,223 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:32,223 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:32,223 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:32,223 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:32,223 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:32,224 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,224 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:32,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:32,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:32,224 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:32,225 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:32,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:32,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:32,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:32,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:32,226 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:32,226 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:32,226 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:32,226 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:32,226 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:32,226 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:32,226 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:32,227 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 14:55:32,227 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:32,227 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:32,227 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:32,227 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:32,227 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:32,227 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:32,239 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304133baf81510f36484daea047196a9a7fe7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133baf81510f36484daea047196a9a7fe7 2023-04-13 14:55:32,613 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/91d5f82b24dd4f0db24117a252ecedbc as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/91d5f82b24dd4f0db24117a252ecedbc 2023-04-13 14:55:32,618 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 7763c91103ebbe310342746ed8aa8aa1/B of 7763c91103ebbe310342746ed8aa8aa1 into 91d5f82b24dd4f0db24117a252ecedbc(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:32,618 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:32,618 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., storeName=7763c91103ebbe310342746ed8aa8aa1/B, priority=9, startTime=1681397732174; duration=0sec 2023-04-13 14:55:32,619 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:32,619 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 14:55:32,625 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 82968 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 14:55:32,626 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7763c91103ebbe310342746ed8aa8aa1/C is initiating minor compaction (all files) 2023-04-13 14:55:32,626 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7763c91103ebbe310342746ed8aa8aa1/C in TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:32,626 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4a63b8a39b3b426e97adbe9b0d752640, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/48f89914110445edacfa4b63188f199c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4b7a2a600243450281fe8948b7fb2def, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b30fdf92042b4559a7e65d6433e879a4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/a17bfb3c9f6b49529f76dcba50723c4f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/8557edaf32b54516a42ac91a27c91d32, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/e7151b18e6754ff0be65b7d4201e93c5] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp, totalSize=81.0 K 2023-04-13 14:55:32,626 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 4a63b8a39b3b426e97adbe9b0d752640, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681397723230 2023-04-13 14:55:32,627 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 48f89914110445edacfa4b63188f199c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1681397723477 2023-04-13 14:55:32,627 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 4b7a2a600243450281fe8948b7fb2def, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681397724853 2023-04-13 14:55:32,627 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting b30fdf92042b4559a7e65d6433e879a4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1681397726054 2023-04-13 14:55:32,628 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting a17bfb3c9f6b49529f76dcba50723c4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681397727307 2023-04-13 14:55:32,628 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 8557edaf32b54516a42ac91a27c91d32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1681397728451 2023-04-13 14:55:32,629 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting e7151b18e6754ff0be65b7d4201e93c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1681397729740 2023-04-13 14:55:32,659 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7763c91103ebbe310342746ed8aa8aa1#C#compaction#359 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:32,664 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=397, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/b589c598f692484f9c800c34487b5be8 2023-04-13 14:55:33,028 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/2e0cf4bb05c747e2be237d8aca50f0a3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/2e0cf4bb05c747e2be237d8aca50f0a3 2023-04-13 14:55:33,032 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 7763c91103ebbe310342746ed8aa8aa1/A of 7763c91103ebbe310342746ed8aa8aa1 into 2e0cf4bb05c747e2be237d8aca50f0a3(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:33,033 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:33,033 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., storeName=7763c91103ebbe310342746ed8aa8aa1/A, priority=9, startTime=1681397732174; duration=0sec 2023-04-13 14:55:33,033 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:33,090 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/11459d9821ab4e6fa46537dbc1435efb as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/11459d9821ab4e6fa46537dbc1435efb 2023-04-13 14:55:33,095 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 7763c91103ebbe310342746ed8aa8aa1/C of 7763c91103ebbe310342746ed8aa8aa1 into 11459d9821ab4e6fa46537dbc1435efb(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:33,095 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:33,095 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., storeName=7763c91103ebbe310342746ed8aa8aa1/C, priority=9, startTime=1681397732174; duration=0sec 2023-04-13 14:55:33,095 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:33,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:33,098 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:33,100 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/83ae6efcd2904955a1ac4ba9257b9e2d 2023-04-13 14:55:33,134 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/4fb73fb1e8504304875f2052f6c69c2c 2023-04-13 14:55:33,140 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/b589c598f692484f9c800c34487b5be8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/b589c598f692484f9c800c34487b5be8 2023-04-13 14:55:33,145 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/b589c598f692484f9c800c34487b5be8, entries=100, sequenceid=397, filesize=18.6 K 2023-04-13 14:55:33,146 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/83ae6efcd2904955a1ac4ba9257b9e2d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/83ae6efcd2904955a1ac4ba9257b9e2d 2023-04-13 14:55:33,151 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/83ae6efcd2904955a1ac4ba9257b9e2d, entries=100, sequenceid=397, filesize=9.5 K 2023-04-13 14:55:33,152 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/4fb73fb1e8504304875f2052f6c69c2c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4fb73fb1e8504304875f2052f6c69c2c 2023-04-13 14:55:33,158 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4fb73fb1e8504304875f2052f6c69c2c, entries=100, sequenceid=397, filesize=9.5 K 2023-04-13 14:55:33,158 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=100.63 KB/103050 for 7763c91103ebbe310342746ed8aa8aa1 in 932ms, sequenceid=397, compaction requested=false 2023-04-13 14:55:33,158 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:33,158 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:33,158 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:33,158 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:33,159 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,160 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:33,160 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:33,162 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,162 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,162 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,162 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:33,163 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 14:55:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,165 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:33,165 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:33,165 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:33,165 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:33,165 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:33,165 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:33,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:33,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25b1a023[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:33,167 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:33,167 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,168 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,169 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:33,169 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:33,169 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:33,169 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,169 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:33,172 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:33,172 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,172 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:33,172 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:33,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:33,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:33,177 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:33,177 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:33,177 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:33,177 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:33,177 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:33,178 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,178 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,178 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:33,179 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,184 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,187 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,187 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:33,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:33,187 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:33,187 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:33,187 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,187 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,187 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,187 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,187 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,187 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:33,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:33,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:33,188 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:33,188 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:33,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:33,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:33,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397793196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:33,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397793197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,288 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:33,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:33,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:33,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:33,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:33,292 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:33,292 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:33,292 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:33,292 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:33,293 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,293 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,295 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:33,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:33,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:33,295 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,295 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,295 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:33,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:33,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:33,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:33,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:33,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:33,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:33,300 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:33,300 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:33,300 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:33,300 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:33,301 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:33,301 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:33,301 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:33,301 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,301 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:33,301 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:33,302 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,302 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,302 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,302 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:33,305 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:33,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1057bec4[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:33,305 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,306 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,306 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:33,306 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:33,306 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:33,306 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,306 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:33,307 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:33,308 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:33,308 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:33,308 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:33,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397793304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:33,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:33,312 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:33,312 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:33,312 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:33,312 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:33,312 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:33,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:33,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397793304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,314 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,314 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,314 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:33,315 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,315 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,318 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:33,318 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:33,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:33,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:33,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:33,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:33,319 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,319 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,319 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,319 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,319 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:33,319 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,319 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,319 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:33,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:33,320 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:33,320 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:33,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:33,420 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:33,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:33,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:33,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:33,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:33,430 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:33,430 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:33,430 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:33,430 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:33,430 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,431 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,432 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:33,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:33,432 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:33,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:33,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:33,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:33,434 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:33,434 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,434 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:33,434 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:33,434 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:33,435 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:33,435 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:33,438 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:33,438 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:33,438 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:33,438 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:33,439 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:33,439 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:33,439 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,439 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:33,439 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:33,444 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,444 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,444 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,444 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:33,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:33,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30a597d3[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:33,446 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:33,446 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,450 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,450 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:33,450 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:33,450 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:33,450 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,450 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:33,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:33,452 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:33,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:33,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:33,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:33,455 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:33,455 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:33,455 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:33,455 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:33,455 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:33,456 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,457 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,457 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:33,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:33,458 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,460 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:33,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:33,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:33,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:33,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:33,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:33,462 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:33,462 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:33,462 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:33,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:33,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:33,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,463 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:33,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397793511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:33,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397793514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,562 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:33,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:33,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:33,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:33,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:33,566 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:33,566 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:33,566 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:33,566 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:33,566 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,566 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,568 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:33,568 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:33,568 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,568 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:33,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:33,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:33,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:33,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:33,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:33,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:33,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:33,574 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:33,574 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:33,574 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:33,574 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:33,574 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:33,574 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:33,574 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,574 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:33,574 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:33,578 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,578 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,579 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,579 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:33,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7067b8b2[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:33,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:33,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,583 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,583 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:33,583 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:33,583 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:33,583 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,583 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:33,586 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:33,586 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:33,586 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,586 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:33,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:33,588 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:33,588 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:33,588 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:33,588 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:33,588 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:33,589 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,589 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,590 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:33,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,590 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,591 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,593 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:33,593 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:33,593 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,594 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:33,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:33,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:33,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:33,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:33,596 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:33,596 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:33,607 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304136bf6c5be1d8242299a0d1723b3643591 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136bf6c5be1d8242299a0d1723b3643591 2023-04-13 14:55:33,616 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=420, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/7674970e827d4fd09916754df13a4ff2 2023-04-13 14:55:33,626 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/43e02393abba4df99508262a0af5cfaa 2023-04-13 14:55:33,696 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:33,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:33,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:33,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:33,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:33,698 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:33,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:33,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:33,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:33,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,700 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:33,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:33,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:33,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,701 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:33,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:33,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:33,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:33,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:33,701 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:33,701 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:33,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:33,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:33,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:33,702 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:33,702 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:33,702 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:33,702 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:33,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:33,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:33,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:33,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:33,704 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,705 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,705 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:33,706 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:33,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58ffae25[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:33,706 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,707 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,707 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:33,707 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,707 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:33,707 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:33,707 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:33,708 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:33,708 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:33,708 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,708 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:33,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:33,711 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:33,711 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:33,711 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:33,711 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:33,711 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:33,712 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,712 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:33,712 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:33,713 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,714 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:33,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:33,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:33,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:33,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:33,717 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:33,717 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:33,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:33,716 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:33,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:33,717 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,717 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,717 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,717 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,717 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,817 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:33,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:33,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:33,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:33,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:33,821 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:33,821 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:33,821 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:33,821 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:33,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:33,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57344 deadline: 1681397793820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:33,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57312 deadline: 1681397793820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,823 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,823 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,824 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:33,824 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:33,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,825 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,825 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:33,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:33,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:33,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:33,826 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:33,826 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:33,827 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:33,827 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:33,827 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:33,827 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:33,827 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:33,828 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:33,828 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:33,828 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:33,828 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:33,828 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,828 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:33,828 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:33,830 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,830 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:33,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75f38a60[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:33,834 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:33,834 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,835 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:33,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:33,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:33,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:33,839 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,840 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:33,840 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:33,840 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:33,840 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:33,840 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:33,840 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:33,840 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:33,842 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,842 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,842 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,847 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:33,851 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:33,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:33,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:33,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:33,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:33,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:33,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,852 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:33,852 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:33,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:33,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:33,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:33,952 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:33,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:33,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:33,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:33,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:33,957 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:33,957 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:33,957 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:33,957 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:33,957 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,957 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,958 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:33,958 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:33,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:33,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:33,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:33,960 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:33,960 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:33,960 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:33,960 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:33,960 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:33,960 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:33,960 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:33,960 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:33,960 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:33,960 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:33,960 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:33,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:33,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:33,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,963 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:33,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:33,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69749002[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:33,964 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:33,964 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,965 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,965 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:33,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:33,966 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,966 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:33,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:33,970 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:33,970 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:33,970 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,970 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:33,972 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:33,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:33,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:33,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:33,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:33,972 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:33,974 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,974 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:33,974 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:33,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,975 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:33,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,976 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:33,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:33,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:33,978 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:33,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:33,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:33,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:33,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:33,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:33,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:33,978 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:33,978 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:33,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:33,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:33,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:34,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/1dbafdca4d9c40f19c567e2ace0f7e27 2023-04-13 14:55:34,041 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/7674970e827d4fd09916754df13a4ff2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/7674970e827d4fd09916754df13a4ff2 2023-04-13 14:55:34,044 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/7674970e827d4fd09916754df13a4ff2, entries=200, sequenceid=420, filesize=32.3 K 2023-04-13 14:55:34,045 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/43e02393abba4df99508262a0af5cfaa as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/43e02393abba4df99508262a0af5cfaa 2023-04-13 14:55:34,049 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/43e02393abba4df99508262a0af5cfaa, entries=150, sequenceid=420, filesize=11.9 K 2023-04-13 14:55:34,049 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/1dbafdca4d9c40f19c567e2ace0f7e27 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/1dbafdca4d9c40f19c567e2ace0f7e27 2023-04-13 14:55:34,053 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/1dbafdca4d9c40f19c567e2ace0f7e27, entries=150, sequenceid=420, filesize=11.9 K 2023-04-13 14:55:34,054 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 7763c91103ebbe310342746ed8aa8aa1 in 892ms, sequenceid=420, compaction requested=true 2023-04-13 14:55:34,054 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:34,054 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:34,054 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:34,054 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:34,054 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:34,054 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:34,055 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78778 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:34,055 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:34,055 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7763c91103ebbe310342746ed8aa8aa1/A is initiating minor compaction (all files) 2023-04-13 14:55:34,055 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7763c91103ebbe310342746ed8aa8aa1/B is initiating minor compaction (all files) 2023-04-13 14:55:34,055 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7763c91103ebbe310342746ed8aa8aa1/A in TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:34,055 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7763c91103ebbe310342746ed8aa8aa1/B in TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:34,055 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/2e0cf4bb05c747e2be237d8aca50f0a3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/b589c598f692484f9c800c34487b5be8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/7674970e827d4fd09916754df13a4ff2] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp, totalSize=76.9 K 2023-04-13 14:55:34,055 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/91d5f82b24dd4f0db24117a252ecedbc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/83ae6efcd2904955a1ac4ba9257b9e2d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/43e02393abba4df99508262a0af5cfaa] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp, totalSize=33.9 K 2023-04-13 14:55:34,056 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 2e0cf4bb05c747e2be237d8aca50f0a3, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1681397729740 2023-04-13 14:55:34,056 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 91d5f82b24dd4f0db24117a252ecedbc, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1681397729740 2023-04-13 14:55:34,056 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting b589c598f692484f9c800c34487b5be8, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1681397730921 2023-04-13 14:55:34,056 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 83ae6efcd2904955a1ac4ba9257b9e2d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1681397730921 2023-04-13 14:55:34,056 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 7674970e827d4fd09916754df13a4ff2, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681397733139 2023-04-13 14:55:34,056 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 43e02393abba4df99508262a0af5cfaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681397733141 2023-04-13 14:55:34,061 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7763c91103ebbe310342746ed8aa8aa1#B#compaction#365 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:34,064 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7763c91103ebbe310342746ed8aa8aa1#A#compaction#366 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:34,079 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:34,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:34,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:34,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:34,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:34,082 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:34,082 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:34,082 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:34,082 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:34,083 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:34,083 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:34,084 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:34,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:34,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:34,084 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,085 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,085 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:34,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:34,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:34,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:34,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:34,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:34,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:34,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:34,086 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:34,086 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:34,086 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:34,086 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:34,086 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:34,086 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. started... 2023-04-13 14:55:34,086 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 14:55:34,086 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:34,086 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:34,086 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:34,086 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:34,086 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:34,086 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:34,100 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041357bce541ff1947eea71ff8761bff05e0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041357bce541ff1947eea71ff8761bff05e0 2023-04-13 14:55:34,121 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=432, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/e4036bc89dab494b9e43b3c7d3de1dc9 2023-04-13 14:55:34,195 DEBUG [Thread-2247] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10377b50 to 127.0.0.1:62626 2023-04-13 14:55:34,196 DEBUG [Thread-2247] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:34,198 DEBUG [Thread-2251] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a35a28d to 127.0.0.1:62626 2023-04-13 14:55:34,199 DEBUG [Thread-2251] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:34,200 DEBUG [Thread-2253] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a1d4246 to 127.0.0.1:62626 2023-04-13 14:55:34,200 DEBUG [Thread-2253] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:34,200 DEBUG [Thread-2249] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4366fcdd to 127.0.0.1:62626 2023-04-13 14:55:34,200 DEBUG [Thread-2249] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:34,203 DEBUG [Thread-2245] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe1db6a to 127.0.0.1:62626 2023-04-13 14:55:34,203 DEBUG [Thread-2245] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:34,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:34,327 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. as already flushing 2023-04-13 14:55:34,328 DEBUG [Thread-2234] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41507d16 to 127.0.0.1:62626 2023-04-13 14:55:34,328 DEBUG [Thread-2234] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:34,329 DEBUG [Thread-2240] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x419b69bf to 127.0.0.1:62626 2023-04-13 14:55:34,330 DEBUG [Thread-2240] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:34,474 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/0a27ab5abb034b9d9e0eff35b29b67f6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/0a27ab5abb034b9d9e0eff35b29b67f6 2023-04-13 14:55:34,477 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7763c91103ebbe310342746ed8aa8aa1/B of 7763c91103ebbe310342746ed8aa8aa1 into 0a27ab5abb034b9d9e0eff35b29b67f6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:34,477 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:34,478 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., storeName=7763c91103ebbe310342746ed8aa8aa1/B, priority=13, startTime=1681397734054; duration=0sec 2023-04-13 14:55:34,478 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:34,478 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:34,478 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:34,479 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7763c91103ebbe310342746ed8aa8aa1/C is initiating minor compaction (all files) 2023-04-13 14:55:34,479 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7763c91103ebbe310342746ed8aa8aa1/C in TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:34,479 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/11459d9821ab4e6fa46537dbc1435efb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4fb73fb1e8504304875f2052f6c69c2c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/1dbafdca4d9c40f19c567e2ace0f7e27] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp, totalSize=33.9 K 2023-04-13 14:55:34,479 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 11459d9821ab4e6fa46537dbc1435efb, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1681397729740 2023-04-13 14:55:34,479 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 4fb73fb1e8504304875f2052f6c69c2c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1681397730921 2023-04-13 14:55:34,479 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 1dbafdca4d9c40f19c567e2ace0f7e27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681397733141 2023-04-13 14:55:34,484 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7763c91103ebbe310342746ed8aa8aa1#C#compaction#369 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:34,487 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/6a7eff0d34634a2e8967660f44c59062 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/6a7eff0d34634a2e8967660f44c59062 2023-04-13 14:55:34,490 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7763c91103ebbe310342746ed8aa8aa1/A of 7763c91103ebbe310342746ed8aa8aa1 into 6a7eff0d34634a2e8967660f44c59062(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:34,490 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:34,490 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., storeName=7763c91103ebbe310342746ed8aa8aa1/A, priority=13, startTime=1681397734054; duration=0sec 2023-04-13 14:55:34,490 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:34,538 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/329f2687c6a74b82aa3c8d379576201b 2023-04-13 14:55:34,892 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/e1ad5fc7da4d42af888e85801bcc62f5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/e1ad5fc7da4d42af888e85801bcc62f5 2023-04-13 14:55:34,895 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7763c91103ebbe310342746ed8aa8aa1/C of 7763c91103ebbe310342746ed8aa8aa1 into e1ad5fc7da4d42af888e85801bcc62f5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:34,895 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:34,895 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1., storeName=7763c91103ebbe310342746ed8aa8aa1/C, priority=13, startTime=1681397734054; duration=0sec 2023-04-13 14:55:34,895 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:34,946 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/b7d340d724854ffe860c737b52eb4ef2 2023-04-13 14:55:34,950 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/e4036bc89dab494b9e43b3c7d3de1dc9 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/e4036bc89dab494b9e43b3c7d3de1dc9 2023-04-13 14:55:34,954 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/e4036bc89dab494b9e43b3c7d3de1dc9, entries=150, sequenceid=432, filesize=25.4 K 2023-04-13 14:55:34,954 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/329f2687c6a74b82aa3c8d379576201b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/329f2687c6a74b82aa3c8d379576201b 2023-04-13 14:55:34,958 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/329f2687c6a74b82aa3c8d379576201b, entries=150, sequenceid=432, filesize=11.9 K 2023-04-13 14:55:34,959 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/b7d340d724854ffe860c737b52eb4ef2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b7d340d724854ffe860c737b52eb4ef2 2023-04-13 14:55:34,963 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b7d340d724854ffe860c737b52eb4ef2, entries=150, sequenceid=432, filesize=11.9 K 2023-04-13 14:55:34,964 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=13.42 KB/13740 for 7763c91103ebbe310342746ed8aa8aa1 in 878ms, sequenceid=432, compaction requested=false 2023-04-13 14:55:34,964 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:34,964 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:34,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:34,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:34,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:34,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:34,965 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,965 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:34,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:34,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:34,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:34,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@241b07b6[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:34,967 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:34,967 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:34,968 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:34,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:34,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:34,968 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,968 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:34,968 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:34,970 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:34,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:34,972 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:34,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:34,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:34,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:34,972 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:34,975 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:34,975 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:34,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:34,975 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:34,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:34,975 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:34,976 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:34,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:34,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:34,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:34,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:34,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:34,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:34,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:34,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:34,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:34,979 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:34,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:34,979 DEBUG [Thread-2244] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:34,979 DEBUG [Thread-2244] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:34,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:34,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:35,079 DEBUG [Thread-2244] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:35,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:36,578 DEBUG [Thread-2236] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49b5bd0d to 127.0.0.1:62626 2023-04-13 14:55:36,578 DEBUG [Thread-2236] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:36,650 DEBUG [Thread-2242] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c6cac96 to 127.0.0.1:62626 2023-04-13 14:55:36,650 DEBUG [Thread-2242] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:36,656 DEBUG [Thread-2238] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c8d927d to 127.0.0.1:62626 2023-04-13 14:55:36,656 DEBUG [Thread-2238] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:36,656 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 14:55:36,657 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 146 2023-04-13 14:55:36,657 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 22 2023-04-13 14:55:36,657 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 23 2023-04-13 14:55:36,657 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 147 2023-04-13 14:55:36,657 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 21 2023-04-13 14:55:36,657 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 14:55:36,657 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 14:55:36,657 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(402): scanned 1422 2023-04-13 14:55:36,657 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(403): verified 4264 rows 2023-04-13 14:55:36,657 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(402): scanned 1425 2023-04-13 14:55:36,657 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(403): verified 4275 rows 2023-04-13 14:55:36,657 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(402): scanned 1413 2023-04-13 14:55:36,657 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(403): verified 4239 rows 2023-04-13 14:55:36,657 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(402): scanned 1418 2023-04-13 14:55:36,657 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(403): verified 4254 rows 2023-04-13 14:55:36,658 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(402): scanned 1426 2023-04-13 14:55:36,658 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(403): verified 4276 rows 2023-04-13 14:55:36,658 INFO [Listener at localhost/37101] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 14:55:36,658 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4473f45c to 127.0.0.1:62626 2023-04-13 14:55:36,658 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:55:36,659 INFO [Listener at localhost/37101] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 14:55:36,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 14:55:36,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:36,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 14:55:36,666 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397736666"}]},"ts":"1681397736666"} 2023-04-13 14:55:36,667 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 14:55:36,668 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 14:55:36,669 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7763c91103ebbe310342746ed8aa8aa1, UNASSIGN}] 2023-04-13 14:55:36,670 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7763c91103ebbe310342746ed8aa8aa1, UNASSIGN 2023-04-13 14:55:36,671 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=7763c91103ebbe310342746ed8aa8aa1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:36,671 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397736670"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397736670"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397736670"}]},"ts":"1681397736670"} 2023-04-13 14:55:36,671 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:55:36,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 14:55:36,823 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:36,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7763c91103ebbe310342746ed8aa8aa1, disabling compactions & flushes 2023-04-13 14:55:36,823 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:36,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:36,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. after waiting 0 ms 2023-04-13 14:55:36,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:36,823 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 7763c91103ebbe310342746ed8aa8aa1 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-13 14:55:36,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=A 2023-04-13 14:55:36,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:36,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=B 2023-04-13 14:55:36,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:36,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7763c91103ebbe310342746ed8aa8aa1, store=C 2023-04-13 14:55:36,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:36,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413983b0cbea67e4b128ed6cd4bb9dfae48 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413983b0cbea67e4b128ed6cd4bb9dfae48 2023-04-13 14:55:36,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 14:55:37,242 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=443, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/2ad6faf6f2654c2db27c5101b557fe30 2023-04-13 14:55:37,254 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/83daa7635dae42b282e7f57febe5a502 2023-04-13 14:55:37,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/5d386d91c90348588c0293b6b0c8bb66 2023-04-13 14:55:37,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 14:55:37,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/A/2ad6faf6f2654c2db27c5101b557fe30 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/2ad6faf6f2654c2db27c5101b557fe30 2023-04-13 14:55:37,273 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/2ad6faf6f2654c2db27c5101b557fe30, entries=100, sequenceid=443, filesize=18.6 K 2023-04-13 14:55:37,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/B/83daa7635dae42b282e7f57febe5a502 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/83daa7635dae42b282e7f57febe5a502 2023-04-13 14:55:37,279 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/83daa7635dae42b282e7f57febe5a502, entries=100, sequenceid=443, filesize=9.5 K 2023-04-13 14:55:37,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/.tmp/C/5d386d91c90348588c0293b6b0c8bb66 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/5d386d91c90348588c0293b6b0c8bb66 2023-04-13 14:55:37,285 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/5d386d91c90348588c0293b6b0c8bb66, entries=100, sequenceid=443, filesize=9.5 K 2023-04-13 14:55:37,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 7763c91103ebbe310342746ed8aa8aa1 in 463ms, sequenceid=443, compaction requested=true 2023-04-13 14:55:37,328 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/4c35de3c604946a59adcaaff80dc2f21, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/8f9b834f14f84699b4e838f500842749, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/91166ef7c7d74113a74feced3dcfa4de, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/e1c0c06953c348e18786140e69952c18, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/512e61a72a6741a3a047a8c50f27be17, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/2d588b28c7524ad5a78dfbd8b899b324, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/e1de50f0883346628dc8457e547f299e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/cd235594f022426fb740bac411811466, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/a27b06e47b34446ba704957043008c9e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/28791ce94b4b4d68bd29b06bf734d01d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/ca8477f869da4aabbcd0f970ce834111, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/3ecfed8f6ffe48c9a1ef0ac8507b4a41, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/d1005df7cf754718b7eff189653fe52c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/160c7a7958b340dd899832f2bcba02eb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/0b67dac80ea2401bbf21aee7edf29f93, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/b50c0112c9a643df8a94802736afa556, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/976203fd1e5a4935a4e00de717895738, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/0204162e49434709a49d948d26244ffa, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/d9689f44ee8646788bf22046c569c32a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/2e0cf4bb05c747e2be237d8aca50f0a3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/b589c598f692484f9c800c34487b5be8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/7674970e827d4fd09916754df13a4ff2] to archive 2023-04-13 14:55:37,329 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:55:37,331 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/4c35de3c604946a59adcaaff80dc2f21 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/4c35de3c604946a59adcaaff80dc2f21 2023-04-13 14:55:37,332 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/8f9b834f14f84699b4e838f500842749 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/8f9b834f14f84699b4e838f500842749 2023-04-13 14:55:37,333 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/91166ef7c7d74113a74feced3dcfa4de to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/91166ef7c7d74113a74feced3dcfa4de 2023-04-13 14:55:37,333 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/e1c0c06953c348e18786140e69952c18 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/e1c0c06953c348e18786140e69952c18 2023-04-13 14:55:37,334 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/512e61a72a6741a3a047a8c50f27be17 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/512e61a72a6741a3a047a8c50f27be17 2023-04-13 14:55:37,335 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/2d588b28c7524ad5a78dfbd8b899b324 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/2d588b28c7524ad5a78dfbd8b899b324 2023-04-13 14:55:37,336 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/e1de50f0883346628dc8457e547f299e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/e1de50f0883346628dc8457e547f299e 2023-04-13 14:55:37,337 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/cd235594f022426fb740bac411811466 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/cd235594f022426fb740bac411811466 2023-04-13 14:55:37,337 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/a27b06e47b34446ba704957043008c9e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/a27b06e47b34446ba704957043008c9e 2023-04-13 14:55:37,338 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/28791ce94b4b4d68bd29b06bf734d01d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/28791ce94b4b4d68bd29b06bf734d01d 2023-04-13 14:55:37,339 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/ca8477f869da4aabbcd0f970ce834111 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/ca8477f869da4aabbcd0f970ce834111 2023-04-13 14:55:37,340 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/3ecfed8f6ffe48c9a1ef0ac8507b4a41 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/3ecfed8f6ffe48c9a1ef0ac8507b4a41 2023-04-13 14:55:37,341 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/d1005df7cf754718b7eff189653fe52c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/d1005df7cf754718b7eff189653fe52c 2023-04-13 14:55:37,342 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/160c7a7958b340dd899832f2bcba02eb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/160c7a7958b340dd899832f2bcba02eb 2023-04-13 14:55:37,343 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/0b67dac80ea2401bbf21aee7edf29f93 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/0b67dac80ea2401bbf21aee7edf29f93 2023-04-13 14:55:37,344 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/b50c0112c9a643df8a94802736afa556 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/b50c0112c9a643df8a94802736afa556 2023-04-13 14:55:37,345 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/976203fd1e5a4935a4e00de717895738 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/976203fd1e5a4935a4e00de717895738 2023-04-13 14:55:37,345 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/0204162e49434709a49d948d26244ffa to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/0204162e49434709a49d948d26244ffa 2023-04-13 14:55:37,346 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/d9689f44ee8646788bf22046c569c32a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/d9689f44ee8646788bf22046c569c32a 2023-04-13 14:55:37,347 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/2e0cf4bb05c747e2be237d8aca50f0a3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/2e0cf4bb05c747e2be237d8aca50f0a3 2023-04-13 14:55:37,348 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/b589c598f692484f9c800c34487b5be8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/b589c598f692484f9c800c34487b5be8 2023-04-13 14:55:37,349 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/7674970e827d4fd09916754df13a4ff2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/7674970e827d4fd09916754df13a4ff2 2023-04-13 14:55:37,352 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/ac238953e3924f4cbb9c8054bace2140, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/51e8cb4e0e194422896216d5ded96ecb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/7743183e0d0b43889df07a512fc3e8c4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/7c880a9fd57d4b099f225ab6bb3884cc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/6dae7cc1b75e469393d3d18ff5252bca, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/8a9ab894950d428ab649ffc1f8146982, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/134e4d6db0ae45a88640a3b2724984a2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/e68eee004d0941f391429a2b355f1281, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/48dc3c99f77d4b488dcf26bee09c01fc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/5189738cd7844f0299ac2cdf71c019d6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/de3f25d68f73496d8208d8e796112589, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/6c0aaccc09474165811d7b1f5811b658, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/86536308cd6541f295cadd068b196f32, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/078c27598f7d49e6bf5758037c3057db, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/26a64a9e7fec460c9092e37d3f5bcbc6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/af802aec94bd40c7b8de59443907c8bb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/affacc8e618548dca01ca135816d99cd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/ff9da4465414482582e0c2bd6e9ac3f1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/91d5f82b24dd4f0db24117a252ecedbc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/1f5ff4eb11684f9e9b28780ff7707510, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/83ae6efcd2904955a1ac4ba9257b9e2d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/43e02393abba4df99508262a0af5cfaa] to archive 2023-04-13 14:55:37,352 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:55:37,354 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/ac238953e3924f4cbb9c8054bace2140 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/ac238953e3924f4cbb9c8054bace2140 2023-04-13 14:55:37,355 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/51e8cb4e0e194422896216d5ded96ecb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/51e8cb4e0e194422896216d5ded96ecb 2023-04-13 14:55:37,356 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/7743183e0d0b43889df07a512fc3e8c4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/7743183e0d0b43889df07a512fc3e8c4 2023-04-13 14:55:37,357 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/7c880a9fd57d4b099f225ab6bb3884cc to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/7c880a9fd57d4b099f225ab6bb3884cc 2023-04-13 14:55:37,358 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/6dae7cc1b75e469393d3d18ff5252bca to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/6dae7cc1b75e469393d3d18ff5252bca 2023-04-13 14:55:37,359 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/8a9ab894950d428ab649ffc1f8146982 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/8a9ab894950d428ab649ffc1f8146982 2023-04-13 14:55:37,361 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/134e4d6db0ae45a88640a3b2724984a2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/134e4d6db0ae45a88640a3b2724984a2 2023-04-13 14:55:37,361 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/e68eee004d0941f391429a2b355f1281 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/e68eee004d0941f391429a2b355f1281 2023-04-13 14:55:37,362 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/48dc3c99f77d4b488dcf26bee09c01fc to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/48dc3c99f77d4b488dcf26bee09c01fc 2023-04-13 14:55:37,363 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/5189738cd7844f0299ac2cdf71c019d6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/5189738cd7844f0299ac2cdf71c019d6 2023-04-13 14:55:37,364 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/de3f25d68f73496d8208d8e796112589 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/de3f25d68f73496d8208d8e796112589 2023-04-13 14:55:37,365 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/6c0aaccc09474165811d7b1f5811b658 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/6c0aaccc09474165811d7b1f5811b658 2023-04-13 14:55:37,366 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/86536308cd6541f295cadd068b196f32 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/86536308cd6541f295cadd068b196f32 2023-04-13 14:55:37,367 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/078c27598f7d49e6bf5758037c3057db to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/078c27598f7d49e6bf5758037c3057db 2023-04-13 14:55:37,367 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/26a64a9e7fec460c9092e37d3f5bcbc6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/26a64a9e7fec460c9092e37d3f5bcbc6 2023-04-13 14:55:37,368 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/af802aec94bd40c7b8de59443907c8bb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/af802aec94bd40c7b8de59443907c8bb 2023-04-13 14:55:37,369 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/affacc8e618548dca01ca135816d99cd to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/affacc8e618548dca01ca135816d99cd 2023-04-13 14:55:37,370 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/ff9da4465414482582e0c2bd6e9ac3f1 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/ff9da4465414482582e0c2bd6e9ac3f1 2023-04-13 14:55:37,370 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/91d5f82b24dd4f0db24117a252ecedbc to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/91d5f82b24dd4f0db24117a252ecedbc 2023-04-13 14:55:37,371 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/1f5ff4eb11684f9e9b28780ff7707510 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/1f5ff4eb11684f9e9b28780ff7707510 2023-04-13 14:55:37,372 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/83ae6efcd2904955a1ac4ba9257b9e2d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/83ae6efcd2904955a1ac4ba9257b9e2d 2023-04-13 14:55:37,372 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/43e02393abba4df99508262a0af5cfaa to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/43e02393abba4df99508262a0af5cfaa 2023-04-13 14:55:37,374 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/7a20536cb775409ca10915edefa7e410, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/916a970a04444735996ba588d4cbbac1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b295aa232cba4c058c5cdcc39a784d34, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/59a67e13d77046cf8708833c7adeb2f7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/c3d35ee93b084a1092f35241e53c47c2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/bd9e1311412d4872948d39272fb08c71, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/8f603c721fdb402a85c49ff4bc3239fc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/c5f1a76cad064a5bb5195b7fbd81585a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/abca699ce6c74e63b02c11b7b0f890e5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/430e970922da4c659e0094096c1e5ccd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/ff03f30cd5794974b9982ccb9c767562, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4a63b8a39b3b426e97adbe9b0d752640, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/0b591b582287486cb9f222ebad5bcc1f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/48f89914110445edacfa4b63188f199c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4b7a2a600243450281fe8948b7fb2def, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b30fdf92042b4559a7e65d6433e879a4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/a17bfb3c9f6b49529f76dcba50723c4f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/8557edaf32b54516a42ac91a27c91d32, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/11459d9821ab4e6fa46537dbc1435efb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/e7151b18e6754ff0be65b7d4201e93c5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4fb73fb1e8504304875f2052f6c69c2c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/1dbafdca4d9c40f19c567e2ace0f7e27] to archive 2023-04-13 14:55:37,375 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:55:37,376 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/7a20536cb775409ca10915edefa7e410 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/7a20536cb775409ca10915edefa7e410 2023-04-13 14:55:37,377 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/916a970a04444735996ba588d4cbbac1 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/916a970a04444735996ba588d4cbbac1 2023-04-13 14:55:37,378 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b295aa232cba4c058c5cdcc39a784d34 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b295aa232cba4c058c5cdcc39a784d34 2023-04-13 14:55:37,379 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/59a67e13d77046cf8708833c7adeb2f7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/59a67e13d77046cf8708833c7adeb2f7 2023-04-13 14:55:37,380 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/c3d35ee93b084a1092f35241e53c47c2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/c3d35ee93b084a1092f35241e53c47c2 2023-04-13 14:55:37,380 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/bd9e1311412d4872948d39272fb08c71 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/bd9e1311412d4872948d39272fb08c71 2023-04-13 14:55:37,381 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/8f603c721fdb402a85c49ff4bc3239fc to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/8f603c721fdb402a85c49ff4bc3239fc 2023-04-13 14:55:37,382 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/c5f1a76cad064a5bb5195b7fbd81585a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/c5f1a76cad064a5bb5195b7fbd81585a 2023-04-13 14:55:37,382 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/abca699ce6c74e63b02c11b7b0f890e5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/abca699ce6c74e63b02c11b7b0f890e5 2023-04-13 14:55:37,383 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/430e970922da4c659e0094096c1e5ccd to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/430e970922da4c659e0094096c1e5ccd 2023-04-13 14:55:37,384 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/ff03f30cd5794974b9982ccb9c767562 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/ff03f30cd5794974b9982ccb9c767562 2023-04-13 14:55:37,385 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4a63b8a39b3b426e97adbe9b0d752640 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4a63b8a39b3b426e97adbe9b0d752640 2023-04-13 14:55:37,385 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/0b591b582287486cb9f222ebad5bcc1f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/0b591b582287486cb9f222ebad5bcc1f 2023-04-13 14:55:37,386 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/48f89914110445edacfa4b63188f199c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/48f89914110445edacfa4b63188f199c 2023-04-13 14:55:37,387 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4b7a2a600243450281fe8948b7fb2def to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4b7a2a600243450281fe8948b7fb2def 2023-04-13 14:55:37,388 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b30fdf92042b4559a7e65d6433e879a4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b30fdf92042b4559a7e65d6433e879a4 2023-04-13 14:55:37,389 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/a17bfb3c9f6b49529f76dcba50723c4f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/a17bfb3c9f6b49529f76dcba50723c4f 2023-04-13 14:55:37,389 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/8557edaf32b54516a42ac91a27c91d32 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/8557edaf32b54516a42ac91a27c91d32 2023-04-13 14:55:37,390 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/11459d9821ab4e6fa46537dbc1435efb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/11459d9821ab4e6fa46537dbc1435efb 2023-04-13 14:55:37,391 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/e7151b18e6754ff0be65b7d4201e93c5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/e7151b18e6754ff0be65b7d4201e93c5 2023-04-13 14:55:37,392 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4fb73fb1e8504304875f2052f6c69c2c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/4fb73fb1e8504304875f2052f6c69c2c 2023-04-13 14:55:37,393 DEBUG [StoreCloser-TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/1dbafdca4d9c40f19c567e2ace0f7e27 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/1dbafdca4d9c40f19c567e2ace0f7e27 2023-04-13 14:55:37,397 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/recovered.edits/446.seqid, newMaxSeqId=446, maxSeqId=4 2023-04-13 14:55:37,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1. 2023-04-13 14:55:37,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7763c91103ebbe310342746ed8aa8aa1: 2023-04-13 14:55:37,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:37,401 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=7763c91103ebbe310342746ed8aa8aa1, regionState=CLOSED 2023-04-13 14:55:37,401 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397737401"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397737401"}]},"ts":"1681397737401"} 2023-04-13 14:55:37,403 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-13 14:55:37,403 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 7763c91103ebbe310342746ed8aa8aa1, server=jenkins-hbase4.apache.org,43875,1681397617638 in 731 msec 2023-04-13 14:55:37,405 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-13 14:55:37,405 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7763c91103ebbe310342746ed8aa8aa1, UNASSIGN in 734 msec 2023-04-13 14:55:37,405 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397737405"}]},"ts":"1681397737405"} 2023-04-13 14:55:37,406 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 14:55:37,408 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 14:55:37,409 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 748 msec 2023-04-13 14:55:37,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 14:55:37,769 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-13 14:55:37,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 14:55:37,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:37,772 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:37,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-13 14:55:37,772 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:37,776 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:37,777 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/recovered.edits] 2023-04-13 14:55:37,780 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/2ad6faf6f2654c2db27c5101b557fe30 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/2ad6faf6f2654c2db27c5101b557fe30 2023-04-13 14:55:37,781 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/6a7eff0d34634a2e8967660f44c59062 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/6a7eff0d34634a2e8967660f44c59062 2023-04-13 14:55:37,782 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/e4036bc89dab494b9e43b3c7d3de1dc9 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/A/e4036bc89dab494b9e43b3c7d3de1dc9 2023-04-13 14:55:37,784 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/0a27ab5abb034b9d9e0eff35b29b67f6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/0a27ab5abb034b9d9e0eff35b29b67f6 2023-04-13 14:55:37,785 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/329f2687c6a74b82aa3c8d379576201b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/329f2687c6a74b82aa3c8d379576201b 2023-04-13 14:55:37,785 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/83daa7635dae42b282e7f57febe5a502 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/B/83daa7635dae42b282e7f57febe5a502 2023-04-13 14:55:37,787 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/5d386d91c90348588c0293b6b0c8bb66 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/5d386d91c90348588c0293b6b0c8bb66 2023-04-13 14:55:37,788 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b7d340d724854ffe860c737b52eb4ef2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/b7d340d724854ffe860c737b52eb4ef2 2023-04-13 14:55:37,789 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/e1ad5fc7da4d42af888e85801bcc62f5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/C/e1ad5fc7da4d42af888e85801bcc62f5 2023-04-13 14:55:37,791 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/recovered.edits/446.seqid to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1/recovered.edits/446.seqid 2023-04-13 14:55:37,791 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7763c91103ebbe310342746ed8aa8aa1 2023-04-13 14:55:37,791 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 14:55:37,792 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 14:55:37,792 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-13 14:55:37,794 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041301dc3fcfb46247d1ab1b5afd548e435a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041301dc3fcfb46247d1ab1b5afd548e435a 2023-04-13 14:55:37,795 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041313d55ebf919f4bd5944d9a5185d4c2b4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041313d55ebf919f4bd5944d9a5185d4c2b4 2023-04-13 14:55:37,796 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304131a023faf0bce4de98b4ff01984fd3749 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304131a023faf0bce4de98b4ff01984fd3749 2023-04-13 14:55:37,797 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413265a7615bef3419f8ece10c1db69dc16 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413265a7615bef3419f8ece10c1db69dc16 2023-04-13 14:55:37,798 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133baf81510f36484daea047196a9a7fe7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133baf81510f36484daea047196a9a7fe7 2023-04-13 14:55:37,799 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134a2a3d8f10be47f09f2224e950d10134 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134a2a3d8f10be47f09f2224e950d10134 2023-04-13 14:55:37,800 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134ca8fe855db64651b26d79c8498e073f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134ca8fe855db64651b26d79c8498e073f 2023-04-13 14:55:37,801 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041357bce541ff1947eea71ff8761bff05e0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041357bce541ff1947eea71ff8761bff05e0 2023-04-13 14:55:37,801 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136bf6c5be1d8242299a0d1723b3643591 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136bf6c5be1d8242299a0d1723b3643591 2023-04-13 14:55:37,802 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136e320c685b874873aeaea9fcc8aa10f2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136e320c685b874873aeaea9fcc8aa10f2 2023-04-13 14:55:37,803 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137954c51b5f2c47059807f316c1405e5d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137954c51b5f2c47059807f316c1405e5d 2023-04-13 14:55:37,804 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137cb065b4a4ca4edd81fa7bf7cf33624c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137cb065b4a4ca4edd81fa7bf7cf33624c 2023-04-13 14:55:37,805 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413983b0cbea67e4b128ed6cd4bb9dfae48 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413983b0cbea67e4b128ed6cd4bb9dfae48 2023-04-13 14:55:37,806 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413a963442361164db5b075571fa76ea78a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413a963442361164db5b075571fa76ea78a 2023-04-13 14:55:37,807 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c59e7bcae3a4417e8d8a761752533bec to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c59e7bcae3a4417e8d8a761752533bec 2023-04-13 14:55:37,808 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d4c10ff3ecba49ad8cf90125142f748d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d4c10ff3ecba49ad8cf90125142f748d 2023-04-13 14:55:37,809 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d873f13c283d4022b937748fb4140696 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d873f13c283d4022b937748fb4140696 2023-04-13 14:55:37,810 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e4475d20341543daa07b417ff254e957 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e4475d20341543daa07b417ff254e957 2023-04-13 14:55:37,810 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f64cabdb25bd4471bf115d3d9bd16aef to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f64cabdb25bd4471bf115d3d9bd16aef 2023-04-13 14:55:37,811 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413fb41bdf5c4984eeeb571508d528c3d50 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413fb41bdf5c4984eeeb571508d528c3d50 2023-04-13 14:55:37,812 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 14:55:37,813 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:37,815 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 14:55:37,816 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 14:55:37,817 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:37,817 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 14:55:37,817 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681397737817"}]},"ts":"9223372036854775807"} 2023-04-13 14:55:37,818 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 14:55:37,818 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7763c91103ebbe310342746ed8aa8aa1, NAME => 'TestAcidGuarantees,,1681397711118.7763c91103ebbe310342746ed8aa8aa1.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 14:55:37,818 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 14:55:37,818 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681397737818"}]},"ts":"9223372036854775807"} 2023-04-13 14:55:37,819 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 14:55:37,821 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:37,821 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 51 msec 2023-04-13 14:55:37,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-13 14:55:37,873 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-13 14:55:37,884 INFO [Listener at localhost/37101] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229 (was 227) - Thread LEAK? -, OpenFileDescriptor=538 (was 558), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=421 (was 298) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=662 (was 778) 2023-04-13 14:55:37,895 INFO [Listener at localhost/37101] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229, OpenFileDescriptor=538, MaxFileDescriptor=60000, SystemLoadAverage=421, ProcessCount=169, AvailableMemoryMB=662 2023-04-13 14:55:37,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 14:55:37,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 14:55:37,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 14:55:37,899 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 14:55:37,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-13 14:55:37,900 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 14:55:37,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 14:55:37,901 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:37,901 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0 empty. 2023-04-13 14:55:37,901 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:37,902 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 14:55:38,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 14:55:38,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 14:55:38,309 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 14:55:38,311 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 46dced31cbc65aff91a95c238f6401f0, NAME => 'TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp 2023-04-13 14:55:38,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 14:55:38,726 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:55:38,726 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 46dced31cbc65aff91a95c238f6401f0, disabling compactions & flushes 2023-04-13 14:55:38,726 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:38,726 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:38,726 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. after waiting 0 ms 2023-04-13 14:55:38,726 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:38,727 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:38,727 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:38,729 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 14:55:38,730 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397738729"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397738729"}]},"ts":"1681397738729"} 2023-04-13 14:55:38,731 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 14:55:38,731 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 14:55:38,731 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397738731"}]},"ts":"1681397738731"} 2023-04-13 14:55:38,732 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 14:55:38,735 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=46dced31cbc65aff91a95c238f6401f0, ASSIGN}] 2023-04-13 14:55:38,737 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=46dced31cbc65aff91a95c238f6401f0, ASSIGN 2023-04-13 14:55:38,737 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=46dced31cbc65aff91a95c238f6401f0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43875,1681397617638; forceNewPlan=false, retain=false 2023-04-13 14:55:38,889 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=46dced31cbc65aff91a95c238f6401f0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:38,889 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397738889"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397738889"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397738889"}]},"ts":"1681397738889"} 2023-04-13 14:55:38,890 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:55:39,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 14:55:39,022 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-13 14:55:39,044 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:39,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 46dced31cbc65aff91a95c238f6401f0, NAME => 'TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.', STARTKEY => '', ENDKEY => ''} 2023-04-13 14:55:39,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:39,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:55:39,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:39,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:39,046 INFO [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:39,048 DEBUG [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A 2023-04-13 14:55:39,048 DEBUG [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A 2023-04-13 14:55:39,048 INFO [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:55:39,048 INFO [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 46dced31cbc65aff91a95c238f6401f0 columnFamilyName A 2023-04-13 14:55:39,049 INFO [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] regionserver.HStore(310): Store=46dced31cbc65aff91a95c238f6401f0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:55:39,049 INFO [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:39,050 DEBUG [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B 2023-04-13 14:55:39,050 DEBUG [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B 2023-04-13 14:55:39,050 INFO [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:55:39,050 INFO [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 46dced31cbc65aff91a95c238f6401f0 columnFamilyName B 2023-04-13 14:55:39,051 INFO [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] regionserver.HStore(310): Store=46dced31cbc65aff91a95c238f6401f0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:55:39,051 INFO [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:39,052 DEBUG [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C 2023-04-13 14:55:39,052 DEBUG [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C 2023-04-13 14:55:39,052 INFO [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:55:39,052 INFO [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 46dced31cbc65aff91a95c238f6401f0 columnFamilyName C 2023-04-13 14:55:39,053 INFO [StoreOpener-46dced31cbc65aff91a95c238f6401f0-1] regionserver.HStore(310): Store=46dced31cbc65aff91a95c238f6401f0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:55:39,053 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:39,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:39,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:39,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 14:55:39,056 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:39,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 14:55:39,058 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 46dced31cbc65aff91a95c238f6401f0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59560869, jitterRate=-0.11247389018535614}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 14:55:39,058 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:39,058 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., pid=49, masterSystemTime=1681397739041 2023-04-13 14:55:39,059 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:39,059 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:39,060 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=46dced31cbc65aff91a95c238f6401f0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:39,060 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397739060"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681397739060"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681397739060"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681397739060"}]},"ts":"1681397739060"} 2023-04-13 14:55:39,062 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-13 14:55:39,062 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 in 171 msec 2023-04-13 14:55:39,063 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-13 14:55:39,063 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=46dced31cbc65aff91a95c238f6401f0, ASSIGN in 327 msec 2023-04-13 14:55:39,064 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 14:55:39,064 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397739064"}]},"ts":"1681397739064"} 2023-04-13 14:55:39,065 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 14:55:39,066 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 14:55:39,067 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1690 sec 2023-04-13 14:55:40,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 14:55:40,005 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-13 14:55:40,006 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x2a987907 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:40,012 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2df63a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:40,014 DEBUG [Listener at localhost/37101] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:40,017 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53240, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:40,018 DEBUG [Listener at localhost/37101] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 14:55:40,020 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46472, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 14:55:40,022 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x37705401 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:40,032 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ec943a9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:40,033 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x15880ffb to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:40,041 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20ed9117, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:40,041 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x052203cb to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:40,049 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50002343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:40,050 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x420f945a to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:40,054 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5726fb8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:40,054 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x51b8df2a to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:40,058 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f2b3c03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:40,059 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x70e2be04 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:40,063 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@461f5f3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:40,064 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x067ea274 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:40,069 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63093e21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:40,069 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x78592df0 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:40,072 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b475d8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:40,073 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x659c68ab to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:40,076 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@549bef45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:40,076 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x680e773e to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:55:40,080 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ed183a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:55:40,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:40,094 DEBUG [hconnection-0x2a7665ae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:40,099 DEBUG [hconnection-0x5a130fa7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:40,099 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53246, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:40,099 DEBUG [hconnection-0x68019b50-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:40,100 DEBUG [hconnection-0xeae7bd1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:40,100 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53256, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:40,100 DEBUG [hconnection-0x43b0fe87-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:40,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:40,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:40,109 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:40,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:40,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:40,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:40,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,110 DEBUG [hconnection-0x24e3d1b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:40,110 DEBUG [hconnection-0x30a871e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:40,112 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53284, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:40,112 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53268, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:40,112 DEBUG [hconnection-0x6d22e10e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:40,113 DEBUG [hconnection-0x39f6081c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:40,113 DEBUG [hconnection-0x4f77690b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:55:40,113 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:40,113 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53290, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:40,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:40,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:40,114 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,114 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53304, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:40,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,114 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,114 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:40,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:40,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:40,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:40,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:40,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:40,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:40,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:40,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:40,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:40,116 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:40,116 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:40,117 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:40,117 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:40,118 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:40,118 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:40,118 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,118 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:40,118 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:40,121 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53300, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:40,121 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53296, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:40,121 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:40,121 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53306, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:55:40,125 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,125 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,125 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,125 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:40,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:40,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b00b96[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:40,126 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:40,126 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,127 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:40,128 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,128 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:40,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:40,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:40,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:40,130 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:55:40,131 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:40,131 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:40,131 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:40,131 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:40,131 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:40,131 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:40,132 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:40,132 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,132 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:40,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,132 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:40,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:40,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:40,135 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:40,135 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:40,135 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:40,135 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:40,135 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:40,144 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,146 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,146 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:40,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,146 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:40,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,148 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:40,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,151 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,151 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,151 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,151 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:40,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:40,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:40,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:40,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:40,151 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,151 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:40,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:40,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:40,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:40,151 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:40,151 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:40,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397800155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397800156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397800157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397800157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681397740135, max timestamp=1681397740156Num uniques 859; 2023-04-13 14:55:40,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-13 14:55:40,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681397740135, max timestamp=1681397740156Num uniques 859; 2023-04-13 14:55:40,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-13 14:55:40,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681397740144, max timestamp=1681397740156Num uniques 859; 2023-04-13 14:55:40,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-13 14:55:40,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397800206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,251 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:40,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:40,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:40,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397800257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397800258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397800258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397800260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:40,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:40,271 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:40,271 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:40,271 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:40,271 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:40,272 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,272 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,273 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,273 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:40,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:40,274 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,274 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:40,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:40,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:40,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:40,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:40,276 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:40,276 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:40,276 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:40,282 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:40,282 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:40,282 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:40,282 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:40,282 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:40,282 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:40,282 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:40,282 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:40,282 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,283 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:40,283 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:40,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,285 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,285 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,285 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:40,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:40,287 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:40,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68f9214a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:40,287 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,288 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,288 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:40,288 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:40,288 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:40,289 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,289 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:40,291 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:40,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:40,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:40,294 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:40,294 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:40,294 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:40,294 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:40,294 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:40,296 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,296 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,296 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:40,297 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:40,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:40,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,298 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,299 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,300 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:40,300 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:40,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:40,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:40,300 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:40,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:40,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:40,300 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,300 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,300 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:40,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:40,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:40,300 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,300 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:40,300 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:40,300 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,300 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397800308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,400 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:40,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:40,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:40,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:40,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:40,405 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:40,405 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:40,405 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:40,405 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:40,406 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,406 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,407 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,407 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,407 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:40,407 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:40,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:40,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:40,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:40,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:40,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:40,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:40,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:40,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:40,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:40,415 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:40,415 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:40,415 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:40,416 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:40,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:40,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:40,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:40,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:40,418 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,418 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,418 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,418 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:40,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:40,420 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:40,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@254048d7[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:40,420 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,421 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,421 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:40,421 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:40,421 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:40,421 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:40,422 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:40,422 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:40,422 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,422 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:40,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:40,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:40,426 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:40,426 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:40,426 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:40,426 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:40,426 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:40,427 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,427 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,427 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:40,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:40,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:40,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,429 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,437 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,440 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:40,440 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:40,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:40,441 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,441 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,441 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,441 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,441 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,441 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,441 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:40,441 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:40,448 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:40,448 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:40,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:40,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:40,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:40,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:40,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397800460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397800460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397800460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397800462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397800510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,541 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:40,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:40,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:40,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:40,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:40,545 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:40,545 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:40,545 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:40,545 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:40,545 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,545 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,546 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,546 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:40,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:40,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:40,547 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,547 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:40,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:40,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:40,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:40,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:40,548 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:40,549 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:40,549 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:40,549 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:40,549 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:40,549 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:40,549 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:40,549 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:40,549 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:40,549 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:40,549 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:40,549 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,549 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:40,549 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:40,551 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,551 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,551 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,551 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:40,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:40,553 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:40,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@599acd7e[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:40,553 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,554 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:40,554 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:40,554 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:40,554 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,554 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:40,560 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,561 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:40,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,561 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:40,561 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:40,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:40,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:40,562 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:40,562 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:40,563 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:40,563 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:40,563 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:40,564 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,564 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,564 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:40,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:40,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,565 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:40,565 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/79768685d3344e54bd4b76c60df87be3 2023-04-13 14:55:40,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,572 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,577 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,577 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,577 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,577 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,577 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,577 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,577 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:40,577 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:40,577 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:40,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:40,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:40,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:40,577 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:40,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:40,578 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:40,578 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:40,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:40,678 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:40,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:40,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:40,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:40,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:40,686 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:40,686 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:40,686 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:40,686 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:40,686 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,686 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,687 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,687 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:40,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:40,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:40,688 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,688 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:40,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:40,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:40,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:40,689 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:40,689 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:40,689 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:40,693 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:40,693 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:40,693 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:40,693 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:40,693 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:40,693 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:40,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:40,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:40,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:40,696 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:40,698 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,698 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,699 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,699 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:40,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:40,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58f79160[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:40,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:40,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,701 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:40,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:40,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:40,702 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,702 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:40,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:40,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:40,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:40,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:40,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:40,709 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:40,709 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:40,709 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:40,709 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:40,709 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:40,710 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,711 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,711 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:40,711 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:40,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:40,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,712 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,715 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,715 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,715 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:40,715 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:40,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:40,716 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:40,715 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:40,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:40,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:40,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:40,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:40,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,716 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:40,716 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:40,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:40,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397800765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397800766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397800766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397800766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,821 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:40,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:40,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:40,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:40,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:40,826 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:40,826 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:40,826 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:40,826 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:40,826 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,826 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:40,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397800822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,828 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,828 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:40,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:40,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:40,828 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,828 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:40,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:40,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:40,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:40,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:40,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:40,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:40,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:40,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:40,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:40,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:40,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:40,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:40,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:40,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:40,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:40,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:40,834 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,834 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,836 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:40,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:40,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79250845[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:40,837 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:40,837 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,838 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:40,839 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:40,839 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:40,839 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,839 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:40,842 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:40,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:40,844 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:40,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:40,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:40,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:40,844 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:40,846 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,847 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:40,847 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:40,848 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:40,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:40,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:40,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:40,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,850 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:40,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,852 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:40,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:40,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:40,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:40,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:40,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:40,853 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:40,853 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:40,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:40,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:40,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:40,953 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:40,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:40,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:40,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:40,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:40,959 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:40,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:40,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:40,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:40,960 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,960 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,961 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:40,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:40,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:40,961 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,962 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,962 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:40,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:40,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:40,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:40,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:40,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:40,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:40,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:40,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:40,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:40,966 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:40,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:40,968 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e84e146[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:40,968 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:40,969 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,969 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:40,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:40,970 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:40,971 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:40,971 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:40,971 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,971 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:40,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:40,984 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:40,984 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:40,984 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:40,984 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:40,984 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:40,988 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:40,989 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,988 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/76a10b5424854f10a7f9ca1ae4ae4205 2023-04-13 14:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:40,988 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:40,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:40,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:40,991 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:40,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:40,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,994 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,994 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:40,994 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:40,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:40,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:40,994 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:40,994 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:40,995 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:40,995 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:40,995 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,995 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:40,995 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:40,995 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:40,995 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:40,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:41,095 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:41,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:41,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:41,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:41,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:41,103 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:41,103 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:41,103 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:41,103 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:41,104 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,104 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,105 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:41,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:41,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:41,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:41,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:41,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:41,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:41,107 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,107 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:41,107 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:41,107 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:41,107 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:41,107 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:41,107 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:41,107 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:41,107 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:41,107 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:41,107 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:41,107 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:41,107 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:41,108 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,108 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:41,108 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:41,109 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,109 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:41,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:41,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:41,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:41,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:41,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:41,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:41,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51fa26d5[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:41,113 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:41,113 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,117 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,117 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:41,117 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:41,117 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:41,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:41,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:41,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:41,119 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:41,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:41,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:41,122 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:41,122 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:41,122 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:41,122 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:41,122 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:41,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,124 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:41,124 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:41,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:41,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,127 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:41,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,129 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:41,129 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:41,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:41,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:41,130 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,130 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,130 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,130 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,130 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,130 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,130 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,131 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:41,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:41,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:41,131 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:41,131 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:41,231 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:41,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:41,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:41,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:41,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:41,244 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:41,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:41,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:41,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:41,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,245 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,247 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,247 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,247 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:41,247 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:41,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:41,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:41,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:41,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:41,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:41,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:41,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:41,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:41,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:41,263 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:41,263 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:41,265 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:41,265 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:41,265 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:41,265 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:41,266 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:41,266 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:41,266 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,266 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:41,266 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:41,268 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,268 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:41,268 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,268 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:41,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:41,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:41,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:41,270 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:41,270 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35e2d11b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:41,271 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,271 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:41,271 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,271 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:41,271 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:41,271 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:41,273 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:41,273 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:41,273 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:41,273 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:41,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:41,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:41,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:41,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:41,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:41,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:41,277 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:41,277 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:41,277 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:41,277 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:41,277 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:41,281 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,281 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:41,281 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:41,282 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:41,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:41,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:41,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397801272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:41,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:41,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397801272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397801272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:41,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397801274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,288 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:41,289 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:41,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:41,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:41,289 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:41,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:41,289 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,289 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,289 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,289 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,289 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,289 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,289 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:41,289 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:41,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:41,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:41,289 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:41,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:41,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:41,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397801330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,389 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:41,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:41,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:41,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:41,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:41,392 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:41,392 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:41,393 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:41,393 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:41,393 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,393 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,394 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:41,394 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:41,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:41,394 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:41,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:41,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:41,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:41,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:41,396 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:41,396 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:41,396 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:41,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:41,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:41,400 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:41,400 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:41,400 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:41,400 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:41,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:41,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:41,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:41,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:41,402 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:41,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:41,402 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:41,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:41,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:41,406 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,407 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:41,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:41,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@645480c2[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:41,410 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:41,410 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,412 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:41,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:41,412 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:41,414 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:41,414 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:41,414 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:41,414 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,414 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:41,414 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:41,414 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:41,414 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:41,414 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:41,414 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:41,414 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:41,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:41,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:41,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:41,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:41,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:41,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:41,417 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:41,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,419 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,426 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:41,426 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:41,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:41,426 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:41,426 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,426 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,426 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:41,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:41,426 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,426 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:41,426 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:41,426 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:41,426 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:41,426 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:41,452 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/aef8f87559cc47e1ba2634054d52804b 2023-04-13 14:55:41,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/79768685d3344e54bd4b76c60df87be3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/79768685d3344e54bd4b76c60df87be3 2023-04-13 14:55:41,461 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/79768685d3344e54bd4b76c60df87be3, entries=150, sequenceid=14, filesize=11.6 K 2023-04-13 14:55:41,462 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/76a10b5424854f10a7f9ca1ae4ae4205 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/76a10b5424854f10a7f9ca1ae4ae4205 2023-04-13 14:55:41,476 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/76a10b5424854f10a7f9ca1ae4ae4205, entries=150, sequenceid=14, filesize=11.6 K 2023-04-13 14:55:41,478 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/aef8f87559cc47e1ba2634054d52804b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/aef8f87559cc47e1ba2634054d52804b 2023-04-13 14:55:41,491 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/aef8f87559cc47e1ba2634054d52804b, entries=150, sequenceid=14, filesize=11.6 K 2023-04-13 14:55:41,492 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=122.60 KB/125545 for 46dced31cbc65aff91a95c238f6401f0 in 1362ms, sequenceid=14, compaction requested=false 2023-04-13 14:55:41,492 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-13 14:55:41,493 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:41,527 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:41,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:41,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:41,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:41,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:41,540 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:41,540 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:41,540 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:41,540 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:41,540 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,541 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,542 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,542 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:41,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:41,542 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,543 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:41,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:41,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:41,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:41,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:41,556 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:41,556 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:41,556 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:41,568 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:41,568 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:41,573 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:41,573 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:41,573 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=122.60 KB heapSize=335.24 KB 2023-04-13 14:55:41,573 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:41,574 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:41,574 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:41,574 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:41,574 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:41,574 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:42,001 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/205ba22091474a5eadaa1cb6dccd7cfb 2023-04-13 14:55:42,026 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/03bdb8788a014719a5718b5de4af6507 2023-04-13 14:55:42,082 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.15 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/207be4e558a846ad92036e58d408fb5d 2023-04-13 14:55:42,086 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/205ba22091474a5eadaa1cb6dccd7cfb as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/205ba22091474a5eadaa1cb6dccd7cfb 2023-04-13 14:55:42,090 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/205ba22091474a5eadaa1cb6dccd7cfb, entries=150, sequenceid=46, filesize=11.6 K 2023-04-13 14:55:42,091 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/03bdb8788a014719a5718b5de4af6507 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/03bdb8788a014719a5718b5de4af6507 2023-04-13 14:55:42,120 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/03bdb8788a014719a5718b5de4af6507, entries=150, sequenceid=46, filesize=11.6 K 2023-04-13 14:55:42,121 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/207be4e558a846ad92036e58d408fb5d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/207be4e558a846ad92036e58d408fb5d 2023-04-13 14:55:42,133 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/207be4e558a846ad92036e58d408fb5d, entries=150, sequenceid=46, filesize=11.6 K 2023-04-13 14:55:42,134 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~122.60 KB/125545, heapSize ~335.20 KB/343240, currentSize=0 B/0 for 46dced31cbc65aff91a95c238f6401f0 in 561ms, sequenceid=46, compaction requested=false 2023-04-13 14:55:42,137 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:42,137 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:42,137 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:42,137 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:42,137 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,137 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:42,137 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:42,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:42,139 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:42,140 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,140 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:42,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:42,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1da3d3df[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:42,141 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:42,141 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,142 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,142 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:42,142 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:42,142 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:42,143 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,143 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:42,145 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:42,145 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:42,145 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:42,145 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:42,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:42,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:42,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:42,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:42,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:42,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:42,148 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:42,148 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:42,148 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:42,148 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:42,148 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:42,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:42,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:42,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:42,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:42,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:42,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:42,151 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:42,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,153 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,156 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,156 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,156 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:42,156 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:42,156 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,156 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,156 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,156 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:42,156 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,156 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:42,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:42,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:42,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:42,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:42,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:42,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:42,156 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:42,157 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:42,257 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:42,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:42,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:42,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:42,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:42,264 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:42,264 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:42,264 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:42,264 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:42,265 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,265 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:42,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:42,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:42,266 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,267 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,267 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:42,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:42,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:42,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:42,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:42,284 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:42,284 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:42,284 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:42,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:42,289 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:42,289 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:42,290 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:42,290 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:42,290 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:42,290 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:42,290 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:42,290 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,290 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:42,290 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:42,292 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,292 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:42,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:42,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:42,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:42,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d6f9230[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:42,295 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:42,295 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,296 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,296 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:42,296 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:42,296 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:42,296 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,296 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:42,297 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,297 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:42,298 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:42,298 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:42,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:42,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:42,300 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:42,300 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:42,300 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:42,300 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:42,300 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:42,301 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:55:42,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:42,302 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:42,302 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,302 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,302 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:42,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,304 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,305 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:42,305 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:42,305 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:42,305 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:42,305 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:42,305 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:42,311 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,311 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,311 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:42,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:42,312 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:42,312 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:42,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:42,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:42,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:42,311 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,312 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,312 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,312 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,311 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:42,312 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:42,312 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:42,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:42,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:42,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397742332, max timestamp=1681397742346Num uniques 859; 2023-04-13 14:55:42,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397742332, max timestamp=1681397742346Num uniques 859; 2023-04-13 14:55:42,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:42,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397742332, max timestamp=1681397742346Num uniques 859; 2023-04-13 14:55:42,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:42,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:42,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:42,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397802366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:42,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397802373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:42,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397802374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:42,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397802375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:42,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397802376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,412 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:42,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:42,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:42,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:42,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:42,415 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:42,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:42,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:42,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:42,416 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,416 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,417 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:42,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:42,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:42,418 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,418 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:42,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:42,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:42,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:42,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:42,421 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:42,422 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:42,422 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:42,423 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:42,423 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:42,423 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:42,423 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:42,424 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:42,424 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:42,424 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:42,424 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:42,424 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,424 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:42,424 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:42,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,425 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:42,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:42,426 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,426 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:42,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:42,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:42,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:42,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ea6f28f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:42,427 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:42,427 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,428 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,429 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:42,429 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:42,429 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:42,429 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,429 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:42,431 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,431 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:42,431 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:42,431 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:42,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:42,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:42,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:42,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:42,434 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:42,434 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:42,434 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:42,434 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:42,434 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:42,435 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,435 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:42,435 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:42,436 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,442 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:42,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,446 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,446 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,446 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,446 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,446 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,446 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,446 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,446 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:42,446 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:42,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:42,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:42,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:42,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:42,446 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:42,446 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:42,446 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:42,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:42,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:42,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:42,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397802477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:42,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397802485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:42,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397802485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:42,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397802486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:42,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397802486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,547 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:42,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:42,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:42,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:42,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:42,551 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:42,551 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:42,551 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:42,551 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:42,551 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,551 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,556 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,557 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,557 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:42,557 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:42,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:42,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:42,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:42,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:42,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:42,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:42,568 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:42,568 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:42,568 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:42,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:42,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:42,572 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:42,572 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:42,572 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:42,572 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:42,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:42,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:42,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:42,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:42,575 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,575 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:42,576 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,576 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:42,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:42,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e342709[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:42,577 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:42,577 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,579 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,579 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:42,579 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:42,579 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:42,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:42,581 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,581 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:42,581 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:42,581 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:42,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:42,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:42,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:42,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:42,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:42,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:42,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:42,585 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:42,585 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:42,585 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:42,585 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:42,585 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:42,588 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,588 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:42,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,589 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:42,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:42,590 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:42,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,591 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:42,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:42,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:42,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:42,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:42,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:42,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:42,596 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,596 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,596 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,596 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,596 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,596 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,596 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,596 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:42,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:42,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:42,597 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:42,597 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:42,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:42,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397802686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,697 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:42,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:42,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:42,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:42,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397802695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:42,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397802695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:42,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397802696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:42,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397802698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:42,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:42,710 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:42,710 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:42,710 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:42,710 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:42,711 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,711 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,712 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:42,712 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:42,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:42,713 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,713 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:42,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:42,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:42,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:42,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:42,714 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:42,714 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:42,714 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:42,716 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:42,716 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:42,716 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:42,716 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:42,716 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:42,716 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:42,716 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:42,716 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:42,716 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,716 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:42,716 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:42,720 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,720 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:42,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:42,720 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,720 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:42,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:42,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:42,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:42,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:42,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d0a3e3b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:42,722 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:42,722 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,725 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,725 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:42,725 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:42,725 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:42,725 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,725 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:42,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:42,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:42,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:42,727 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:42,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:42,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:42,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:42,729 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:42,729 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:42,729 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:42,729 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:42,729 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:42,731 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,731 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:42,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,731 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:42,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:42,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:42,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:42,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:42,732 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:42,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,735 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:42,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:42,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,737 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:42,737 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:42,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:42,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:42,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:42,738 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,738 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,738 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,738 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,738 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:42,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:42,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:42,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:42,738 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:42,739 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,739 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:42,739 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:42,778 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/84ad7ead528240578db3249f6f7e7f15 2023-04-13 14:55:42,797 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/d851bfde2c3241919acf87f42d10f77b 2023-04-13 14:55:42,819 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/c3dfcf58eb28489290fb127909aa693f 2023-04-13 14:55:42,823 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/84ad7ead528240578db3249f6f7e7f15 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/84ad7ead528240578db3249f6f7e7f15 2023-04-13 14:55:42,830 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/84ad7ead528240578db3249f6f7e7f15, entries=200, sequenceid=58, filesize=13.9 K 2023-04-13 14:55:42,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/d851bfde2c3241919acf87f42d10f77b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/d851bfde2c3241919acf87f42d10f77b 2023-04-13 14:55:42,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/d851bfde2c3241919acf87f42d10f77b, entries=150, sequenceid=58, filesize=11.6 K 2023-04-13 14:55:42,838 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/c3dfcf58eb28489290fb127909aa693f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c3dfcf58eb28489290fb127909aa693f 2023-04-13 14:55:42,839 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:42,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:42,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:42,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:42,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:42,855 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:42,855 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:42,856 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:42,856 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:42,856 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,856 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,859 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:42,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:42,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:42,859 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c3dfcf58eb28489290fb127909aa693f, entries=150, sequenceid=58, filesize=11.6 K 2023-04-13 14:55:42,860 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,860 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:42,860 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:42,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:42,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:42,862 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 46dced31cbc65aff91a95c238f6401f0 in 561ms, sequenceid=58, compaction requested=true 2023-04-13 14:55:42,863 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:42,863 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:42,863 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:42,863 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:42,863 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:42,863 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:42,864 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:42,865 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/B is initiating minor compaction (all files) 2023-04-13 14:55:42,865 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/B in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:42,865 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/76a10b5424854f10a7f9ca1ae4ae4205, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/03bdb8788a014719a5718b5de4af6507, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/d851bfde2c3241919acf87f42d10f77b] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=34.8 K 2023-04-13 14:55:42,864 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:42,865 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/A is initiating minor compaction (all files) 2023-04-13 14:55:42,865 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/A in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:42,865 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/79768685d3344e54bd4b76c60df87be3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/205ba22091474a5eadaa1cb6dccd7cfb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/84ad7ead528240578db3249f6f7e7f15] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=37.1 K 2023-04-13 14:55:42,865 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 76a10b5424854f10a7f9ca1ae4ae4205, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681397740124 2023-04-13 14:55:42,869 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 03bdb8788a014719a5718b5de4af6507, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681397740167 2023-04-13 14:55:42,871 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 79768685d3344e54bd4b76c60df87be3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681397740124 2023-04-13 14:55:42,871 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting d851bfde2c3241919acf87f42d10f77b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681397742295 2023-04-13 14:55:42,872 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 205ba22091474a5eadaa1cb6dccd7cfb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681397740167 2023-04-13 14:55:42,872 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 84ad7ead528240578db3249f6f7e7f15, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681397742295 2023-04-13 14:55:42,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:42,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:42,880 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:42,880 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:42,880 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:42,888 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:42,888 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:42,888 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:42,888 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:42,888 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-13 14:55:42,889 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:42,889 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:42,889 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:42,889 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:42,889 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:42,889 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:42,897 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#B#compaction#383 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:42,905 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#A#compaction#384 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:42,944 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/37c2b33931054730926f504472264ded as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/37c2b33931054730926f504472264ded 2023-04-13 14:55:42,948 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/B of 46dced31cbc65aff91a95c238f6401f0 into 37c2b33931054730926f504472264ded(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:42,948 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:42,948 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/B, priority=13, startTime=1681397742863; duration=0sec 2023-04-13 14:55:42,948 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:42,948 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:42,951 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 14:55:42,952 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:42,952 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/C is initiating minor compaction (all files) 2023-04-13 14:55:42,952 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/C in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:42,952 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/aef8f87559cc47e1ba2634054d52804b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/207be4e558a846ad92036e58d408fb5d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c3dfcf58eb28489290fb127909aa693f] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=34.8 K 2023-04-13 14:55:42,953 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting aef8f87559cc47e1ba2634054d52804b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681397740124 2023-04-13 14:55:42,953 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 207be4e558a846ad92036e58d408fb5d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681397740167 2023-04-13 14:55:42,954 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting c3dfcf58eb28489290fb127909aa693f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681397742295 2023-04-13 14:55:42,965 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/055bf8ca45494580b91925853ab14c20 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/055bf8ca45494580b91925853ab14c20 2023-04-13 14:55:42,970 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/A of 46dced31cbc65aff91a95c238f6401f0 into 055bf8ca45494580b91925853ab14c20(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:42,970 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:42,970 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/A, priority=13, startTime=1681397742863; duration=0sec 2023-04-13 14:55:42,970 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:42,975 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#C#compaction#386 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:43,016 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:43,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:43,032 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/07ec740fa75d4a1eaa6c23f480676425 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/07ec740fa75d4a1eaa6c23f480676425 2023-04-13 14:55:43,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397803022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,036 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/C of 46dced31cbc65aff91a95c238f6401f0 into 07ec740fa75d4a1eaa6c23f480676425(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:43,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,036 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:43,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397803023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,036 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/C, priority=13, startTime=1681397742863; duration=0sec 2023-04-13 14:55:43,036 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:43,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397803028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397803030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397803031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397803137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397803139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397803142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397803142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397803143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,337 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/e5e8f7df9b12494e8b11a1a52a9b3112 2023-04-13 14:55:43,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397803346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397803349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397803350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397803350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397803352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,365 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/bfd1c8d96185437bb58ba82b8e7ffb3b 2023-04-13 14:55:43,392 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/d37649126e534a1796f2c5604c03689f 2023-04-13 14:55:43,396 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/e5e8f7df9b12494e8b11a1a52a9b3112 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/e5e8f7df9b12494e8b11a1a52a9b3112 2023-04-13 14:55:43,401 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/e5e8f7df9b12494e8b11a1a52a9b3112, entries=150, sequenceid=92, filesize=11.6 K 2023-04-13 14:55:43,402 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/bfd1c8d96185437bb58ba82b8e7ffb3b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/bfd1c8d96185437bb58ba82b8e7ffb3b 2023-04-13 14:55:43,407 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/bfd1c8d96185437bb58ba82b8e7ffb3b, entries=150, sequenceid=92, filesize=11.6 K 2023-04-13 14:55:43,408 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/d37649126e534a1796f2c5604c03689f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/d37649126e534a1796f2c5604c03689f 2023-04-13 14:55:43,414 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/d37649126e534a1796f2c5604c03689f, entries=150, sequenceid=92, filesize=11.6 K 2023-04-13 14:55:43,415 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 46dced31cbc65aff91a95c238f6401f0 in 527ms, sequenceid=92, compaction requested=false 2023-04-13 14:55:43,415 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:43,415 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:43,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:43,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:43,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:43,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:43,419 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:43,419 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:43,420 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:43,420 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:43,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:43,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c95ddb8[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:43,421 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:43,421 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:43,423 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,423 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:43,424 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:43,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:43,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:43,425 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:43,425 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:43,425 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:43,425 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:43,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:43,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:43,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:43,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:43,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:43,428 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:43,428 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:43,428 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:43,428 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:43,428 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:43,431 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:43,431 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:43,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:43,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:43,432 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:43,433 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,435 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,438 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:43,439 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:43,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:43,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:43,439 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,439 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:43,439 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:43,439 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,440 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:43,440 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:43,440 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:43,440 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:43,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:43,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:43,440 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:43,440 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:43,540 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:43,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:43,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:43,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:43,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:43,547 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:43,548 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:43,548 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:43,548 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:43,548 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:43,548 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:43,549 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:43,549 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:43,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:43,550 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,550 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:43,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:43,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:43,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:43,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:43,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:43,556 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:43,556 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:43,556 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:43,561 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:43,561 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:43,561 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:43,561 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:43,561 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 14:55:43,561 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:43,562 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:43,562 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:43,562 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:43,562 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:43,562 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:43,641 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/585c238028fa4fcc98fda04908eee75f 2023-04-13 14:55:43,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:43,661 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:43,686 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/f8711b7405e34711b903822b6166f5a8 2023-04-13 14:55:43,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397743679, max timestamp=1681397743696Num uniques 859; 2023-04-13 14:55:43,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397743679, max timestamp=1681397743696Num uniques 859; 2023-04-13 14:55:43,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397743679, max timestamp=1681397743696Num uniques 859; 2023-04-13 14:55:43,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:43,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:43,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:43,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397803728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397803740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397803742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397803744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397803745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397803847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397803847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397803859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397803860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:43,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:43,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397803861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397804055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397804067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397804068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397804069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397804069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,122 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/faa34179a3fc47fdb1f87131b5b23ae7 2023-04-13 14:55:44,127 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/585c238028fa4fcc98fda04908eee75f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/585c238028fa4fcc98fda04908eee75f 2023-04-13 14:55:44,131 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/585c238028fa4fcc98fda04908eee75f, entries=150, sequenceid=106, filesize=11.6 K 2023-04-13 14:55:44,131 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/f8711b7405e34711b903822b6166f5a8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/f8711b7405e34711b903822b6166f5a8 2023-04-13 14:55:44,157 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/f8711b7405e34711b903822b6166f5a8, entries=150, sequenceid=106, filesize=11.6 K 2023-04-13 14:55:44,158 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/faa34179a3fc47fdb1f87131b5b23ae7 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/faa34179a3fc47fdb1f87131b5b23ae7 2023-04-13 14:55:44,162 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/faa34179a3fc47fdb1f87131b5b23ae7, entries=150, sequenceid=106, filesize=11.6 K 2023-04-13 14:55:44,163 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 46dced31cbc65aff91a95c238f6401f0 in 602ms, sequenceid=106, compaction requested=true 2023-04-13 14:55:44,163 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:44,163 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:44,163 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:44,163 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:44,163 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,164 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:44,164 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:44,169 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,169 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:44,169 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,170 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:44,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:44,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:44,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@152e662f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:44,171 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:44,171 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,172 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,172 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:44,172 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:44,173 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:44,174 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:44,174 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:44,174 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,174 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:44,174 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:44,174 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:44,174 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:44,174 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:44,174 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:44,174 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:44,174 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:44,176 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,176 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,176 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:44,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:44,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:44,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,176 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:44,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:44,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:44,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:44,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:44,177 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:44,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:44,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:44,181 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,181 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:44,181 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:44,181 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:44,181 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:44,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:44,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:44,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:44,181 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,181 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,181 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,181 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:44,181 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:44,181 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:44,182 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:44,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:44,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:44,282 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:44,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:44,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:44,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:44,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:44,290 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:44,290 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:44,290 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:44,291 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:44,291 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,291 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:44,292 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:44,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:44,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:44,292 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:44,293 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,293 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:44,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:44,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:44,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:44,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:44,305 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:44,305 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:44,305 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:44,311 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:44,311 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:44,311 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:44,311 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:44,311 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-13 14:55:44,312 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:44,312 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:44,312 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:44,312 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:44,312 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:44,312 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:44,342 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/5a50fa45fcd3438da1598d54151b486e 2023-04-13 14:55:44,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:44,375 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:44,376 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/399fe877f3e6444797cf534ad0d32fef 2023-04-13 14:55:44,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397804396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397804396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397804396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397804398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397804405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397804509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397804509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397804513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397804519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397804520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397804715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397804715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397804715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397804727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:44,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397804728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,806 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/ffb0506015714422a8fdb4ac052a5b08 2023-04-13 14:55:44,811 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/5a50fa45fcd3438da1598d54151b486e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5a50fa45fcd3438da1598d54151b486e 2023-04-13 14:55:44,814 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5a50fa45fcd3438da1598d54151b486e, entries=150, sequenceid=141, filesize=11.8 K 2023-04-13 14:55:44,815 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/399fe877f3e6444797cf534ad0d32fef as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/399fe877f3e6444797cf534ad0d32fef 2023-04-13 14:55:44,821 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/399fe877f3e6444797cf534ad0d32fef, entries=150, sequenceid=141, filesize=11.8 K 2023-04-13 14:55:44,822 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/ffb0506015714422a8fdb4ac052a5b08 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ffb0506015714422a8fdb4ac052a5b08 2023-04-13 14:55:44,827 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ffb0506015714422a8fdb4ac052a5b08, entries=150, sequenceid=141, filesize=11.8 K 2023-04-13 14:55:44,828 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 46dced31cbc65aff91a95c238f6401f0 in 517ms, sequenceid=141, compaction requested=true 2023-04-13 14:55:44,828 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:44,828 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:44,831 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:44,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:44,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4afd17ec[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:44,834 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:44,834 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,837 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,837 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:44,837 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:44,837 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:44,837 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,837 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:44,839 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:44,839 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,839 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:44,839 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:44,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:44,842 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:44,842 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:44,842 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:44,842 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:44,842 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:44,843 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,843 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:44,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,843 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:44,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,847 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:44,847 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:44,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:44,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:44,848 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,848 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:44,848 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:44,848 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:44,848 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:44,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:44,848 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,848 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:44,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,849 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:44,849 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:44,949 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:44,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:44,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:44,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:44,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:44,952 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:44,952 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:44,952 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:44,952 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:44,952 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,952 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:44,955 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:44,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:44,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:44,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:44,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:44,956 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,956 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:44,957 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:44,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:44,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:44,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:44,958 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:44,958 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:44,958 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:44,958 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:44,958 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:44,958 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:44,958 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:44,958 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 14:55:44,958 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:44,958 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:44,958 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:44,959 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:44,959 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:44,959 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:44,980 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/a86331a6d532427e8f33e2b3aa8a5906 2023-04-13 14:55:44,994 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/fcae84eda8a24637abae383d6180fd6d 2023-04-13 14:55:45,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:45,026 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:45,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397805056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681397745045, max timestamp=1681397745061Num uniques 859; 2023-04-13 14:55:45,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681397745048, max timestamp=1681397745061Num uniques 859; 2023-04-13 14:55:45,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-13 14:55:45,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-13 14:55:45,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681397745048, max timestamp=1681397745061Num uniques 859; 2023-04-13 14:55:45,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-13 14:55:45,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397805085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397805089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397805094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397805095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397805164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397805196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397805196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397805217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397805217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397805375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,415 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397805403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397805404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,418 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/de2c94ea324f4b3c8d45cd8368f0529b 2023-04-13 14:55:45,421 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/a86331a6d532427e8f33e2b3aa8a5906 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/a86331a6d532427e8f33e2b3aa8a5906 2023-04-13 14:55:45,424 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/a86331a6d532427e8f33e2b3aa8a5906, entries=150, sequenceid=152, filesize=11.8 K 2023-04-13 14:55:45,425 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/fcae84eda8a24637abae383d6180fd6d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/fcae84eda8a24637abae383d6180fd6d 2023-04-13 14:55:45,428 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/fcae84eda8a24637abae383d6180fd6d, entries=150, sequenceid=152, filesize=11.8 K 2023-04-13 14:55:45,429 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/de2c94ea324f4b3c8d45cd8368f0529b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/de2c94ea324f4b3c8d45cd8368f0529b 2023-04-13 14:55:45,438 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/de2c94ea324f4b3c8d45cd8368f0529b, entries=150, sequenceid=152, filesize=11.8 K 2023-04-13 14:55:45,439 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 46dced31cbc65aff91a95c238f6401f0 in 481ms, sequenceid=152, compaction requested=true 2023-04-13 14:55:45,439 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:45,439 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:45,439 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:45,439 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:45,439 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,439 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:45,439 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:45,441 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:45,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:45,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,441 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=136.42 KB heapSize=376 KB 2023-04-13 14:55:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:45,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:45,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:45,442 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:45,442 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:45,442 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:45,442 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:45,442 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:45,442 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:45,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:45,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@662a036e[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:45,443 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:45,443 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,444 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:45,445 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,445 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:45,447 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:45,447 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,447 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:45,447 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:45,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:45,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:45,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:45,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:45,450 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:45,450 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:45,450 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:45,450 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:45,450 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:45,455 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/3126bfed280a42488549fd61da9937a5 2023-04-13 14:55:45,457 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,457 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:45,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,457 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:45,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:45,458 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:45,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,459 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:45,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:45,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:45,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:45,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:45,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,462 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:45,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:45,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:45,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,462 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:45,462 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:45,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:45,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:45,471 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/77fcf1da87ce4c15980ea615159029e4 2023-04-13 14:55:45,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397805473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397805482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,562 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:45,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:45,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:45,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:45,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:45,570 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:45,570 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:45,570 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:45,570 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:45,570 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,570 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,571 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:45,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:45,571 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:45,572 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,572 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:45,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:45,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:45,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:45,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:45,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:45,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:45,573 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:45,573 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:45,573 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:45,573 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:45,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:45,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:45,574 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,574 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:45,574 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:45,575 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:45,575 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:45,576 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,576 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:45,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:45,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:45,577 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:45,577 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54e06a0e[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:45,579 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,579 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:45,579 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,579 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:45,579 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:45,579 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:45,585 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:45,585 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:45,585 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:45,585 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:45,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:45,587 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:45,587 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:45,587 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:45,587 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:45,587 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:45,589 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,589 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,589 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:45,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,590 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:45,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,591 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:45,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397805583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:45,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:45,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:45,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:45,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:45,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:45,594 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:45,594 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:45,594 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,595 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:45,595 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,595 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:45,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:45,595 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,595 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397805589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397805689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,695 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:45,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:45,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:45,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:45,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:45,698 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:45,698 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:45,698 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:45,698 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:45,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,699 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,700 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:45,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:45,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:45,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:45,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:45,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:45,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:45,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:45,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:45,702 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:45,702 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:45,702 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:45,702 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:45,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:45,702 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:45,703 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,703 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:45,703 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:45,705 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,705 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:45,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@321971a3[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:45,707 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:45,708 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,709 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,709 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:45,709 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:45,709 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:45,709 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,709 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:45,711 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,712 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:45,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:45,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:45,716 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:45,716 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:45,716 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:45,716 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:45,716 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:45,717 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,718 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,718 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:45,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:45,719 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,720 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397805718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,723 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:45,724 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:45,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:45,724 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397805718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,724 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:45,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:45,724 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,724 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,724 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:45,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:45,724 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,725 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,725 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,725 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:45,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:45,725 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:45,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397805793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:45,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397805798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,825 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:45,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:45,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:45,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:45,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:45,828 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:45,828 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:45,829 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:45,829 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:45,829 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,829 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,830 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,830 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:45,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:45,831 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,831 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:45,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:45,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:45,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:45,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:45,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:45,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:45,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:45,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:45,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:45,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:45,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:45,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:45,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:45,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,833 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:45,833 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:45,837 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,837 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:45,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:45,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:45,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:45,839 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:45,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f9427b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:45,839 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,840 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:45,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:45,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:45,840 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,841 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:45,841 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,842 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:45,842 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:45,842 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:45,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:45,844 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:45,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:45,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:45,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:45,844 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:45,847 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,847 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:45,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:45,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:45,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,848 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:45,848 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,850 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:45,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:45,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,853 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:45,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:45,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:45,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:45,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:45,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:45,853 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:45,853 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:45,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:45,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:45,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:45,886 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/7e6d971a60fd47efb47db488de0cc916 2023-04-13 14:55:45,890 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/3126bfed280a42488549fd61da9937a5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3126bfed280a42488549fd61da9937a5 2023-04-13 14:55:45,896 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3126bfed280a42488549fd61da9937a5, entries=250, sequenceid=187, filesize=16.4 K 2023-04-13 14:55:45,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/77fcf1da87ce4c15980ea615159029e4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/77fcf1da87ce4c15980ea615159029e4 2023-04-13 14:55:45,900 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/77fcf1da87ce4c15980ea615159029e4, entries=150, sequenceid=187, filesize=11.8 K 2023-04-13 14:55:45,901 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/7e6d971a60fd47efb47db488de0cc916 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/7e6d971a60fd47efb47db488de0cc916 2023-04-13 14:55:45,904 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/7e6d971a60fd47efb47db488de0cc916, entries=150, sequenceid=187, filesize=11.8 K 2023-04-13 14:55:45,905 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~395.88 KB/405376, currentSize=40.25 KB/41220 for 46dced31cbc65aff91a95c238f6401f0 in 464ms, sequenceid=187, compaction requested=true 2023-04-13 14:55:45,905 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:45,905 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:45,905 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:45,905 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 14:55:45,905 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 14:55:45,905 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:45,906 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76649 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 14:55:45,906 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71869 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 14:55:45,906 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/A is initiating minor compaction (all files) 2023-04-13 14:55:45,906 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/B is initiating minor compaction (all files) 2023-04-13 14:55:45,906 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/A in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:45,906 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/B in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:45,907 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/055bf8ca45494580b91925853ab14c20, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/e5e8f7df9b12494e8b11a1a52a9b3112, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/585c238028fa4fcc98fda04908eee75f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5a50fa45fcd3438da1598d54151b486e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/a86331a6d532427e8f33e2b3aa8a5906, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3126bfed280a42488549fd61da9937a5] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=74.9 K 2023-04-13 14:55:45,907 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/37c2b33931054730926f504472264ded, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/bfd1c8d96185437bb58ba82b8e7ffb3b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/f8711b7405e34711b903822b6166f5a8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/399fe877f3e6444797cf534ad0d32fef, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/fcae84eda8a24637abae383d6180fd6d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/77fcf1da87ce4c15980ea615159029e4] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=70.2 K 2023-04-13 14:55:45,907 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 055bf8ca45494580b91925853ab14c20, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681397742295 2023-04-13 14:55:45,907 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 37c2b33931054730926f504472264ded, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681397742295 2023-04-13 14:55:45,907 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting e5e8f7df9b12494e8b11a1a52a9b3112, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681397742373 2023-04-13 14:55:45,908 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting bfd1c8d96185437bb58ba82b8e7ffb3b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681397742373 2023-04-13 14:55:45,908 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 585c238028fa4fcc98fda04908eee75f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1681397743017 2023-04-13 14:55:45,908 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting f8711b7405e34711b903822b6166f5a8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1681397743017 2023-04-13 14:55:45,908 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 5a50fa45fcd3438da1598d54151b486e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1681397743739 2023-04-13 14:55:45,908 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 399fe877f3e6444797cf534ad0d32fef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1681397743739 2023-04-13 14:55:45,908 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting a86331a6d532427e8f33e2b3aa8a5906, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681397744393 2023-04-13 14:55:45,909 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting fcae84eda8a24637abae383d6180fd6d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681397744393 2023-04-13 14:55:45,909 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 3126bfed280a42488549fd61da9937a5, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681397745084 2023-04-13 14:55:45,909 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 77fcf1da87ce4c15980ea615159029e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681397745084 2023-04-13 14:55:45,920 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#A#compaction#401 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:45,922 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#B#compaction#402 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:45,949 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/600a1a33df314420879345947dcae407 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/600a1a33df314420879345947dcae407 2023-04-13 14:55:45,953 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/A of 46dced31cbc65aff91a95c238f6401f0 into 600a1a33df314420879345947dcae407(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:45,953 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:45,953 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/A, priority=10, startTime=1681397745905; duration=0sec 2023-04-13 14:55:45,953 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:45,953 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 14:55:45,953 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:45,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:45,958 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71869 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 14:55:45,958 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/C is initiating minor compaction (all files) 2023-04-13 14:55:45,958 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/C in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:45,958 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/07ec740fa75d4a1eaa6c23f480676425, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/d37649126e534a1796f2c5604c03689f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/faa34179a3fc47fdb1f87131b5b23ae7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ffb0506015714422a8fdb4ac052a5b08, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/de2c94ea324f4b3c8d45cd8368f0529b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/7e6d971a60fd47efb47db488de0cc916] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=70.2 K 2023-04-13 14:55:45,959 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 07ec740fa75d4a1eaa6c23f480676425, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681397742295 2023-04-13 14:55:45,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:45,959 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting d37649126e534a1796f2c5604c03689f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681397742373 2023-04-13 14:55:45,959 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting faa34179a3fc47fdb1f87131b5b23ae7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1681397743017 2023-04-13 14:55:45,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:45,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:45,960 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:45,960 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting ffb0506015714422a8fdb4ac052a5b08, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1681397743739 2023-04-13 14:55:45,960 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:45,960 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:45,960 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:45,961 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting de2c94ea324f4b3c8d45cd8368f0529b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681397744393 2023-04-13 14:55:45,961 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,961 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,961 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 7e6d971a60fd47efb47db488de0cc916, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681397745084 2023-04-13 14:55:45,962 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,962 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:45,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:45,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:45,962 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:45,962 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:45,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:45,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:45,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:45,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:45,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:45,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:45,965 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:45,965 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:45,965 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:45,965 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:45,965 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:45,965 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 14:55:45,965 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:45,966 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:45,966 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:45,966 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:45,966 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:45,966 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:45,974 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#C#compaction#404 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:45,988 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/2de13d159da948e5b9716d9c4ef384f2 2023-04-13 14:55:45,995 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/6b5800b6807b4198a8cb1c833fdd1789 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/6b5800b6807b4198a8cb1c833fdd1789 2023-04-13 14:55:46,000 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/C of 46dced31cbc65aff91a95c238f6401f0 into 6b5800b6807b4198a8cb1c833fdd1789(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:46,000 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:46,000 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/C, priority=10, startTime=1681397745905; duration=0sec 2023-04-13 14:55:46,000 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:46,006 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/444e1242a6e8463095279c8176faad34 2023-04-13 14:55:46,016 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/5ba3b7f326584cdbae6169eaffe88947 2023-04-13 14:55:46,021 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/2de13d159da948e5b9716d9c4ef384f2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/2de13d159da948e5b9716d9c4ef384f2 2023-04-13 14:55:46,025 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/2de13d159da948e5b9716d9c4ef384f2, entries=150, sequenceid=197, filesize=11.8 K 2023-04-13 14:55:46,025 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-13 14:55:46,025 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/444e1242a6e8463095279c8176faad34 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/444e1242a6e8463095279c8176faad34 2023-04-13 14:55:46,028 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/444e1242a6e8463095279c8176faad34, entries=150, sequenceid=197, filesize=11.8 K 2023-04-13 14:55:46,029 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/5ba3b7f326584cdbae6169eaffe88947 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5ba3b7f326584cdbae6169eaffe88947 2023-04-13 14:55:46,037 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5ba3b7f326584cdbae6169eaffe88947, entries=150, sequenceid=197, filesize=11.8 K 2023-04-13 14:55:46,038 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 46dced31cbc65aff91a95c238f6401f0 in 73ms, sequenceid=197, compaction requested=false 2023-04-13 14:55:46,038 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:46,038 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:46,043 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,043 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,044 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,044 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:46,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26fe1279[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:46,047 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:46,047 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,048 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,048 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:46,048 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:46,048 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:46,049 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,049 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:46,051 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,051 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:46,051 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,051 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:46,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:46,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:46,053 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:46,053 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:46,053 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:46,053 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:46,053 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:46,057 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,057 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,057 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:46,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,058 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,059 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,061 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,061 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,061 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,061 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:46,061 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,061 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:46,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:46,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:46,061 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,061 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,061 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:46,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:46,061 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:46,061 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:46,061 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:46,119 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:55:46,119 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:46,119 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:46,119 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:46,119 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:46,119 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:46,119 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:46,161 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:46,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:46,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:46,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:46,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:46,165 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:46,165 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:46,165 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:46,165 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:46,166 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,166 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,168 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:46,168 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:46,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:46,169 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,169 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:46,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:46,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:46,171 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:46,171 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:46,172 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:46,180 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:46,180 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:46,180 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:46,180 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:46,180 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:46,180 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:46,180 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:46,180 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:46,180 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,180 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:46,180 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:46,183 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,183 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397746151, max timestamp=1681397746182Num uniques 859; 2023-04-13 14:55:46,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:46,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397746151, max timestamp=1681397746182Num uniques 859; 2023-04-13 14:55:46,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397746151, max timestamp=1681397746182Num uniques 859; 2023-04-13 14:55:46,184 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,185 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:46,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:46,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:46,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7199acb2[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:46,188 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:46,188 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,189 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,189 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:46,189 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,189 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:46,189 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:46,189 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:46,190 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:46,190 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,190 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,190 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:46,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:46,198 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:46,198 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:46,198 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:46,198 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:46,198 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:46,200 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,200 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,201 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:46,201 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,202 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:46,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:46,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:46,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,205 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:46,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:46,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:46,205 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:46,205 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:46,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:46,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:46,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397806228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:46,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397806229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:46,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397806233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:46,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397806236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:46,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397806237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,306 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:46,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:46,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:46,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:46,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:46,309 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:46,309 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:46,309 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:46,309 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:46,310 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,310 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,311 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:46,311 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:46,311 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,311 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:46,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:46,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:46,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:46,312 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:46,312 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:46,312 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:46,312 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:46,312 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:46,312 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:46,312 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:46,312 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:46,312 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:46,312 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:46,313 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:46,313 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,313 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:46,313 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:46,314 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,314 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,314 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,314 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:46,316 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:46,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8f45f4b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:46,316 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,318 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,318 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:46,318 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:46,318 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:46,318 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,318 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:46,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:46,319 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:46,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:46,322 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:46,322 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:46,322 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:46,322 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:46,322 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:46,323 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,323 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,323 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:46,324 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,325 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,328 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:46,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:46,328 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,328 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,328 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,328 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,328 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:46,328 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:46,328 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:46,328 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:46,328 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:46,328 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,328 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:46,348 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/e743378e48284f7aa157f646c4bf2f0a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/e743378e48284f7aa157f646c4bf2f0a 2023-04-13 14:55:46,351 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/B of 46dced31cbc65aff91a95c238f6401f0 into e743378e48284f7aa157f646c4bf2f0a(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:46,351 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:46,351 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/B, priority=10, startTime=1681397745905; duration=0sec 2023-04-13 14:55:46,352 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:46,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:46,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397806341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:46,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397806347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:46,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397806347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,428 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:46,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:46,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:46,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:46,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:46,432 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:46,432 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:46,432 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:46,432 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:46,432 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,432 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,434 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:46,434 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:46,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:46,435 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,435 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:46,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:46,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:46,436 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:46,437 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:46,437 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:46,437 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:46,437 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:46,437 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:46,439 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,439 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,439 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,439 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:46,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52443eb4[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:46,441 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:46,441 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,442 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,442 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:46,442 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:46,442 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:46,443 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,444 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:46,445 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,445 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:46,445 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,445 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:46,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:46,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:46,447 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:46,447 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:46,447 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:46,447 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:46,447 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:46,450 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,451 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,451 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:46,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,452 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,460 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,464 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,465 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,465 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,465 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,465 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,465 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,465 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,465 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:46,465 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:46,465 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:46,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:46,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:46,466 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:46,466 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:46,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/0c8ff876e6e444878322803ed945c98b 2023-04-13 14:55:46,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:46,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397806554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:46,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397806555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:46,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397806555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,566 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:46,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:46,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:46,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:46,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:46,569 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:46,569 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:46,569 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:46,569 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:46,570 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,570 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,571 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,571 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:46,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:46,571 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,571 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:46,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:46,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:46,572 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:46,573 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:46,573 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:46,573 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:46,573 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:46,573 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:46,575 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,575 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,576 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,576 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47f935f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:46,577 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:46,577 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,579 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,580 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:46,580 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:46,580 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:46,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:46,581 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:46,581 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,581 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,581 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:46,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:46,585 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:46,585 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:46,585 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:46,585 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:46,585 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:46,586 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,587 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,587 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:46,587 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,588 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,591 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,591 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:46,591 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,591 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,591 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,591 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,591 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,591 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,591 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:46,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:46,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:46,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:46,591 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:46,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:46,592 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:46,592 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:46,692 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:46,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:46,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:46,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:46,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:46,696 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:46,696 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:46,696 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:46,697 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:46,697 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,697 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,702 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:46,702 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:46,703 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,703 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:46,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:46,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:46,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:46,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:46,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:46,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:46,704 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:46,704 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:46,704 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:46,704 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:46,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:46,705 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:46,705 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,705 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:46,705 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:46,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:46,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@383f435e[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:46,708 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:46,708 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,709 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,709 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:46,709 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:46,709 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:46,710 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,710 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,711 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:46,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:46,713 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:46,713 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:46,713 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:46,713 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:46,713 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:46,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,717 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,717 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,718 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,720 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:46,721 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:46,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:46,721 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,721 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,721 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,721 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,721 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,721 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,721 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,722 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:46,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:46,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:46,722 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:46,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:46,722 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:46,822 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:46,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:46,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:46,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:46,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:46,826 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:46,826 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:46,826 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:46,826 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:46,827 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,827 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,828 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:46,828 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:46,828 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,828 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:46,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:46,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:46,829 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:46,829 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:46,830 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:46,830 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:46,830 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:46,830 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:46,832 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,833 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,833 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,833 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:46,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@121b4d15[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:46,834 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:46,834 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,836 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,837 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:46,837 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:46,837 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:46,837 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,837 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:46,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:46,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,838 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:46,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:46,841 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:46,841 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:46,841 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:46,841 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:46,841 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:46,843 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,843 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,843 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,844 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:46,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:46,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,849 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:46,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:46,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:46,850 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:46,850 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:46,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:46,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:46,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397806862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:46,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:46,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397806862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397806862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,936 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/9821c823c49443b0acc82b8fc43b0db3 2023-04-13 14:55:46,950 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:46,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:46,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:46,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:46,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:46,953 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:46,953 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:46,953 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:46,953 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:46,955 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,955 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,959 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:46,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:46,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:46,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:46,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:46,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:46,961 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,961 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:46,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:46,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:46,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:46,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:46,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:46,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:46,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:46,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:46,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:46,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:46,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:46,965 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,965 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,965 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,965 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:46,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6159607c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:46,967 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:46,967 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:46,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:46,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:46,969 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,969 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:46,971 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:46,971 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,971 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,971 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:46,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:46,975 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:46,975 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:46,975 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:46,976 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:46,976 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:46,977 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,977 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:46,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:46,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:46,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:46,979 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:46,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,982 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:46,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,984 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:46,984 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:46,984 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:46,984 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:46,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:46,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:46,984 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,984 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:46,984 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:46,984 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,984 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:46,984 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:46,984 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:46,985 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:46,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:47,085 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:47,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:47,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:47,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:47,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:47,088 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:47,089 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,089 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:47,089 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:47,089 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,089 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,090 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:47,090 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:47,091 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,091 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:47,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:47,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:47,092 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:47,092 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,092 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:47,092 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:47,092 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:47,092 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,092 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:47,093 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:47,093 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,093 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:47,093 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:47,093 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,093 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:47,093 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:47,101 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,101 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,101 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,101 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:47,103 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:47,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19c4f263[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:47,103 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,104 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:47,106 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:47,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,106 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:47,106 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:47,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:47,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:47,106 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:47,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:47,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:47,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:47,106 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:47,108 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,108 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:47,108 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,110 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:47,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:47,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:47,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:47,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:47,114 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,115 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,115 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,115 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,114 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:47,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:47,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:47,116 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:47,116 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:47,216 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:47,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:47,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:47,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:47,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:47,219 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:47,219 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,219 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:47,219 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:47,219 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,220 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,221 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:47,221 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:47,221 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,221 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:47,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:47,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:47,222 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:47,222 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,222 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:47,222 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:47,222 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:47,222 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,223 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:47,223 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:47,223 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,223 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:47,223 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:47,223 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,223 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:47,223 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:47,225 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,225 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,225 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,225 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:47,226 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:47,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b2377b6[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:47,226 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,227 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,227 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:47,227 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:47,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,227 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:47,228 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:47,230 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,230 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:47,230 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,230 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:47,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:47,232 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:47,232 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:47,232 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:47,232 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:47,232 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:47,233 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,233 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,233 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:47,234 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,235 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,237 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:47,237 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:47,237 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,237 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,237 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,237 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:47,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:47,238 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:47,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:47,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:47,238 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:47,238 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:47,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:47,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:47,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:47,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397807238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:47,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397807241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,338 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:47,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:47,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:47,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:47,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:47,342 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:47,342 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,342 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:47,342 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:47,342 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,342 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,343 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:47,343 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:47,344 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,344 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:47,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:47,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/0790b2dec1024044a06496a0e1d76f06 2023-04-13 14:55:47,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:47,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:47,347 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:47,347 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,347 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:47,347 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:47,347 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:47,347 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,347 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:47,347 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:47,347 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,347 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:47,347 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:47,347 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,347 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:47,347 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:47,351 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/0c8ff876e6e444878322803ed945c98b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/0c8ff876e6e444878322803ed945c98b 2023-04-13 14:55:47,352 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,352 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,352 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,352 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:47,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1398ea22[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:47,355 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:47,355 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/0c8ff876e6e444878322803ed945c98b, entries=200, sequenceid=208, filesize=14.1 K 2023-04-13 14:55:47,356 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,356 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/9821c823c49443b0acc82b8fc43b0db3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/9821c823c49443b0acc82b8fc43b0db3 2023-04-13 14:55:47,357 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,357 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:47,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,357 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:47,357 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:47,357 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:47,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:47,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:47,359 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:47,359 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/9821c823c49443b0acc82b8fc43b0db3, entries=100, sequenceid=208, filesize=9.4 K 2023-04-13 14:55:47,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:47,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:47,361 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:47,362 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:47,362 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:47,362 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:47,362 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:47,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/0790b2dec1024044a06496a0e1d76f06 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0790b2dec1024044a06496a0e1d76f06 2023-04-13 14:55:47,365 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,365 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,365 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,366 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0790b2dec1024044a06496a0e1d76f06, entries=100, sequenceid=208, filesize=9.4 K 2023-04-13 14:55:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,368 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,368 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 46dced31cbc65aff91a95c238f6401f0 in 1249ms, sequenceid=208, compaction requested=true 2023-04-13 14:55:47,368 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,368 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:47,368 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:47,368 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:47,368 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:47,368 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,369 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38802 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:47,369 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:47,369 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/B is initiating minor compaction (all files) 2023-04-13 14:55:47,369 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/A is initiating minor compaction (all files) 2023-04-13 14:55:47,370 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/B in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,370 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/A in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,370 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/e743378e48284f7aa157f646c4bf2f0a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/444e1242a6e8463095279c8176faad34, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/9821c823c49443b0acc82b8fc43b0db3] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=33.2 K 2023-04-13 14:55:47,370 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/600a1a33df314420879345947dcae407, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/2de13d159da948e5b9716d9c4ef384f2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/0c8ff876e6e444878322803ed945c98b] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=37.9 K 2023-04-13 14:55:47,370 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting e743378e48284f7aa157f646c4bf2f0a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681397745084 2023-04-13 14:55:47,370 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 600a1a33df314420879345947dcae407, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681397745084 2023-04-13 14:55:47,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:47,371 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-13 14:55:47,371 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 2de13d159da948e5b9716d9c4ef384f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681397745469 2023-04-13 14:55:47,371 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:47,371 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 444e1242a6e8463095279c8176faad34, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681397745469 2023-04-13 14:55:47,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:47,371 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:47,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:47,371 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:47,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:47,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:47,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:47,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:47,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:47,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:47,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,372 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:47,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:47,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:47,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,373 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,373 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:47,373 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 0c8ff876e6e444878322803ed945c98b, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681397746112 2023-04-13 14:55:47,373 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:47,373 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:47,373 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 9821c823c49443b0acc82b8fc43b0db3, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681397746112 2023-04-13 14:55:47,380 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#B#compaction#411 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:47,384 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#A#compaction#412 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:47,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:47,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397807381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:47,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397807384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:47,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397807390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,408 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/0d5c2e91523e42fd80da88ac59f6a834 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/0d5c2e91523e42fd80da88ac59f6a834 2023-04-13 14:55:47,412 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/B of 46dced31cbc65aff91a95c238f6401f0 into 0d5c2e91523e42fd80da88ac59f6a834(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:47,412 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:47,412 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/B, priority=13, startTime=1681397747368; duration=0sec 2023-04-13 14:55:47,412 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:47,412 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:47,413 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:47,413 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/C is initiating minor compaction (all files) 2023-04-13 14:55:47,413 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/C in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,413 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/6b5800b6807b4198a8cb1c833fdd1789, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5ba3b7f326584cdbae6169eaffe88947, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0790b2dec1024044a06496a0e1d76f06] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=33.2 K 2023-04-13 14:55:47,413 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 6b5800b6807b4198a8cb1c833fdd1789, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681397745084 2023-04-13 14:55:47,414 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 5ba3b7f326584cdbae6169eaffe88947, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681397745469 2023-04-13 14:55:47,414 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 0790b2dec1024044a06496a0e1d76f06, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681397746112 2023-04-13 14:55:47,420 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#C#compaction#413 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:47,434 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/1c90f2f5cffe427f87dc531c0e17c3a8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/1c90f2f5cffe427f87dc531c0e17c3a8 2023-04-13 14:55:47,440 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/C of 46dced31cbc65aff91a95c238f6401f0 into 1c90f2f5cffe427f87dc531c0e17c3a8(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:47,440 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:47,440 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/C, priority=13, startTime=1681397747368; duration=0sec 2023-04-13 14:55:47,441 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:47,473 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:47,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:47,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:47,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:47,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:47,477 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:47,477 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,477 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:47,477 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:47,477 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,477 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,478 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:47,478 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:47,479 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,479 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:47,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:47,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,480 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:47,481 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:47,481 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:47,481 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,481 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:47,481 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:47,481 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:47,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:47,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:47,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:47,483 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39e822ad[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:47,485 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:47,485 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,487 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,487 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:47,488 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:47,488 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:47,488 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,488 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:47,489 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,489 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,489 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:47,489 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:47,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397807491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:47,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397807491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:47,494 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:47,495 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:47,495 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:47,495 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:47,495 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:47,496 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,496 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,496 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:47,497 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,498 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,500 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,500 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,500 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,500 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,500 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,500 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:47,500 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:47,500 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:47,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:47,500 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,500 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:47,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:47,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:47,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:47,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:47,501 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:47,501 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:47,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:47,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397807499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,601 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:47,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:47,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:47,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:47,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:47,606 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:47,606 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,606 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:47,606 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:47,606 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,606 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,608 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:47,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,609 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:47,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:47,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:47,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:47,610 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,610 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:47,611 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:47,611 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,611 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:47,611 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:47,611 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,611 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:47,611 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:47,613 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,613 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,614 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,614 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:47,617 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:47,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7018819[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:47,617 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,618 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,618 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:47,618 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:47,618 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,618 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:47,618 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:47,621 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:47,621 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:47,621 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:47,622 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:47,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:47,626 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:47,626 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:47,626 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:47,626 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:47,626 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:47,633 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,633 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,633 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:47,634 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,635 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:47,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:47,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:47,637 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:47,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:47,637 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:47,638 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:47,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:47,637 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:47,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:47,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:47,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397807694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:47,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397807694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:47,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397807704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,738 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:47,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:47,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:47,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:47,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:47,741 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:47,741 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,741 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:47,741 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:47,742 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,742 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,743 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:47,743 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:47,743 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,743 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:47,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:47,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:47,744 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:47,744 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,744 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:47,744 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:47,744 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:47,744 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,744 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:47,745 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:47,745 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,745 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:47,745 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:47,745 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,745 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:47,745 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:47,746 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,746 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,746 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,746 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:47,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47dfd188[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:47,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:47,748 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,752 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:47,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:47,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:47,753 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,753 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:47,756 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,756 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:47,756 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:47,756 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:47,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:47,758 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:47,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:47,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:47,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:47,758 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:47,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,760 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:47,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:47,762 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,763 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:47,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,765 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:47,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:47,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:47,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:47,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:47,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:47,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:47,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,765 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,765 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:47,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:47,765 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:47,795 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/6ef4273160d843249bea6c72dc7f2913 2023-04-13 14:55:47,804 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/73486f2ddd484217b9aaf2d302a01edb 2023-04-13 14:55:47,808 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/74081aff99d144aa81022acb6af20ffd as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/74081aff99d144aa81022acb6af20ffd 2023-04-13 14:55:47,812 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/A of 46dced31cbc65aff91a95c238f6401f0 into 74081aff99d144aa81022acb6af20ffd(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:47,812 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:47,812 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/A, priority=13, startTime=1681397747368; duration=0sec 2023-04-13 14:55:47,813 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:47,866 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:47,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:47,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:47,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:47,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:47,868 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:47,869 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,869 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:47,869 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:47,869 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,869 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,871 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:47,871 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:47,871 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,871 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:47,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:47,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:47,872 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:47,873 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,873 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:47,873 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:47,873 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:47,874 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,874 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,875 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,875 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:47,876 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:47,876 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1342eaab[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:47,877 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,877 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:47,877 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:47,877 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:47,877 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,877 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:47,878 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,878 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:47,878 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,878 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:47,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:47,881 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:47,881 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:47,881 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:47,881 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:47,881 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:47,883 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,883 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:47,883 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:47,884 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,885 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,887 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,887 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:47,887 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:47,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:47,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:47,887 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:47,887 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:47,887 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:47,887 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:47,887 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:47,887 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,887 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,887 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,887 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:47,987 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:47,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:47,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:47,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:47,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:47,991 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:47,991 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,991 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:47,991 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:47,991 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,991 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,993 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,993 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:47,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:47,993 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,993 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:47,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:47,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:47,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:47,994 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:47,994 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:47,994 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:47,994 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:47,994 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:47,994 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,995 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:47,995 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:47,995 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:47,996 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,996 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:47,997 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:47,997 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:47,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:47,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b475f24[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:47,998 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:47,998 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:48,000 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:48,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:48,000 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,001 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:48,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:48,001 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:48,001 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:48,002 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:48,002 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:48,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,002 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:48,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53268 deadline: 1681397807999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:48,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397807999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:48,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:48,005 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:48,005 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:48,005 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:48,005 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:48,005 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:48,007 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,007 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:48,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,007 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:48,007 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:48,009 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:48,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:48,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:48,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:48,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:48,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:48,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:48,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:48,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:48,011 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:48,011 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:48,012 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:48,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:48,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:48,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:48,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:48,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53300 deadline: 1681397808009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,112 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:48,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:48,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:48,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:48,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:48,115 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:48,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:48,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:48,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:48,116 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,116 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:48,118 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:48,118 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:48,118 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,118 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:48,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:48,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:48,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:48,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:48,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:48,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:48,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:48,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:48,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:48,120 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:48,120 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:48,120 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:48,120 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:48,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:48,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:48,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:48,120 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:48,122 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:48,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:48,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@600d4315[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:48,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:48,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:48,126 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:48,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:48,126 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:48,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:48,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:48,126 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:48,127 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:48,130 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:48,130 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:48,130 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:48,130 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:48,130 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:48,135 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,135 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,135 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:48,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:48,135 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:48,137 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,138 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,138 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:48,138 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:48,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:48,139 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:48,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:48,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:48,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:48,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:48,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:48,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:48,139 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:48,139 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:48,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:48,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:48,139 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,221 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/5d5c75c4d9764384865107dbf47e35f2 2023-04-13 14:55:48,225 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/6ef4273160d843249bea6c72dc7f2913 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6ef4273160d843249bea6c72dc7f2913 2023-04-13 14:55:48,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6ef4273160d843249bea6c72dc7f2913, entries=200, sequenceid=245, filesize=14.1 K 2023-04-13 14:55:48,229 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/73486f2ddd484217b9aaf2d302a01edb as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/73486f2ddd484217b9aaf2d302a01edb 2023-04-13 14:55:48,232 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/73486f2ddd484217b9aaf2d302a01edb, entries=150, sequenceid=245, filesize=11.8 K 2023-04-13 14:55:48,233 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/5d5c75c4d9764384865107dbf47e35f2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5d5c75c4d9764384865107dbf47e35f2 2023-04-13 14:55:48,235 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5d5c75c4d9764384865107dbf47e35f2, entries=150, sequenceid=245, filesize=11.8 K 2023-04-13 14:55:48,236 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 46dced31cbc65aff91a95c238f6401f0 in 865ms, sequenceid=245, compaction requested=false 2023-04-13 14:55:48,236 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:48,239 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:48,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:48,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:48,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:48,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:48,242 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:48,242 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:48,243 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:48,243 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:48,243 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,243 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:48,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,244 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:48,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:48,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:48,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:48,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:48,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:48,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:48,245 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:48,245 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:48,245 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:48,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:48,248 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:48,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:48,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:48,248 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 14:55:48,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:48,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:48,248 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:48,249 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:48,249 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:48,249 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:48,281 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/15698f856129465f84ca792a067f2a45 2023-04-13 14:55:48,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:48,510 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:48,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397748561, max timestamp=1681397748578Num uniques 859; 2023-04-13 14:55:48,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:48,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397748561, max timestamp=1681397748578Num uniques 859; 2023-04-13 14:55:48,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:48,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397748561, max timestamp=1681397748578Num uniques 859; 2023-04-13 14:55:48,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:48,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:48,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397808626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:48,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397808627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397748603, max timestamp=1681397748635Num uniques 1018; 2023-04-13 14:55:48,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397748603, max timestamp=1681397748635Num uniques 1018; 2023-04-13 14:55:48,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 14:55:48,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 14:55:48,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397748603, max timestamp=1681397748635Num uniques 1018; 2023-04-13 14:55:48,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 14:55:48,696 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/c73d5550ae984c21964ee4f4d6ff5e44 2023-04-13 14:55:48,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:48,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397808757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:48,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681397748742, max timestamp=1681397748763Num uniques 1027; 2023-04-13 14:55:48,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397808758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 14:55:48,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681397748742, max timestamp=1681397748763Num uniques 1027; 2023-04-13 14:55:48,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 14:55:48,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681397748742, max timestamp=1681397748763Num uniques 1027; 2023-04-13 14:55:48,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 14:55:48,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:48,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397808867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681397748824, max timestamp=1681397748871Num uniques 1036; 2023-04-13 14:55:48,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-13 14:55:48,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681397748824, max timestamp=1681397748871Num uniques 1036; 2023-04-13 14:55:48,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-13 14:55:48,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681397748824, max timestamp=1681397748871Num uniques 1036; 2023-04-13 14:55:48,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-13 14:55:48,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:48,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397808920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:48,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:48,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397808924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:49,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397809026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:49,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397809028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:49,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397809074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,111 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/15a71c86b5474276a8277d2dd9b86175 2023-04-13 14:55:49,114 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/15698f856129465f84ca792a067f2a45 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/15698f856129465f84ca792a067f2a45 2023-04-13 14:55:49,117 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/15698f856129465f84ca792a067f2a45, entries=150, sequenceid=257, filesize=11.8 K 2023-04-13 14:55:49,118 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/c73d5550ae984c21964ee4f4d6ff5e44 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/c73d5550ae984c21964ee4f4d6ff5e44 2023-04-13 14:55:49,122 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/c73d5550ae984c21964ee4f4d6ff5e44, entries=150, sequenceid=257, filesize=11.8 K 2023-04-13 14:55:49,122 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/15a71c86b5474276a8277d2dd9b86175 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/15a71c86b5474276a8277d2dd9b86175 2023-04-13 14:55:49,125 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/15a71c86b5474276a8277d2dd9b86175, entries=150, sequenceid=257, filesize=11.8 K 2023-04-13 14:55:49,126 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 46dced31cbc65aff91a95c238f6401f0 in 878ms, sequenceid=257, compaction requested=true 2023-04-13 14:55:49,126 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:49,126 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,129 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:49,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:49,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e79d0d[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:49,131 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:49,131 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,132 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:49,132 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,132 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:49,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:49,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:49,133 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:49,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:49,136 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:49,136 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:49,136 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:49,136 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:49,136 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:49,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:49,137 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:49,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:49,138 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,140 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,143 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,143 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,143 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:49,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:49,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:49,143 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:49,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:49,143 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:49,143 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,143 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,143 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,143 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:49,143 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:49,143 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,143 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:49,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:49,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:49,233 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=147.60 KB heapSize=418.31 KB 2023-04-13 14:55:49,234 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:49,234 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:55:49,234 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:49,234 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:55:49,234 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:49,234 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:55:49,242 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/23c20183717b4979988eb569fe99e6f4 2023-04-13 14:55:49,244 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:49,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:49,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:49,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:49,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:49,247 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:49,247 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:49,247 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:49,247 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:49,248 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,248 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,250 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:49,250 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:49,251 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,251 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:49,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:49,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:49,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:49,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:49,252 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:49,253 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:49,253 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:49,253 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:49,253 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:49,253 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:49,253 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:49,253 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:49,253 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:49,253 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/b9203e949afe47f5bbe7e3224f6d21b7 2023-04-13 14:55:49,253 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,253 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:49,253 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:49,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:49,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397809245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,255 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:49,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:49,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397809254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:49,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5df43328[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:49,259 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:49,259 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,262 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,262 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:49,262 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:49,262 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,262 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:49,262 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:49,264 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:49,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:49,267 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:49,267 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:49,267 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:49,267 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:49,267 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:49,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:49,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397809261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:49,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397809261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,270 DEBUG [Thread-2916] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:55:49,270 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,270 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:49,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,271 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:49,272 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:49,272 DEBUG [Thread-2924] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4186 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:55:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,273 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,275 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,275 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,275 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:49,275 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:49,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:49,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:49,275 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:49,276 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:49,276 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,276 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,276 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,276 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,276 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:49,276 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:49,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:49,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:49,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:49,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397809354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:49,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397809360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,376 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:49,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:49,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:49,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:49,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:49,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:49,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397809379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,380 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:49,380 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:49,380 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:49,380 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:49,380 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,380 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,381 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:49,381 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:49,382 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,382 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:49,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:49,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:49,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:49,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:49,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:49,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:49,383 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:49,383 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:49,383 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:49,384 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:49,384 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:49,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:49,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:49,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:49,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:49,386 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,386 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:49,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:49,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:49,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:49,389 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:49,389 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c9bf9cd[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:49,390 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,390 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:49,390 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:49,390 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:49,390 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,390 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:49,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:49,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:49,391 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:49,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:49,394 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:49,394 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:49,394 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:49,394 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:49,394 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:49,395 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,395 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:49,395 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:49,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:49,397 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,398 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:49,403 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:49,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:49,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:49,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:49,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:49,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:49,404 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:49,404 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:49,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:49,504 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:49,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:49,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:49,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:49,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:49,508 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:49,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:49,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:49,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:49,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,509 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:49,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:49,510 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,510 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:49,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:49,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:49,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:49,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:49,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:49,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:49,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:49,511 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:49,511 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:49,511 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:49,511 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:49,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:49,512 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:49,512 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,512 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:49,512 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:49,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,514 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:49,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:49,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:49,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60416a0[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:49,516 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:49,516 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,522 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:49,524 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:49,524 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:49,524 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,524 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:49,524 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:49,524 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:49,524 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:49,524 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:49,524 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:49,524 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:49,524 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:49,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:49,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:49,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:49,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:49,530 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,532 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:49,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:49,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:49,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:49,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,533 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,533 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,533 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,533 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:49,533 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:49,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:49,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:49,533 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:49,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:49,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:49,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:49,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:49,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397809560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:49,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397809567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,633 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:49,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:49,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:49,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:49,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:49,636 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:49,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:49,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:49,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:49,637 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,637 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,639 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:49,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,640 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,640 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,640 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:49,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:49,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:49,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:49,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:49,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:49,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:49,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:49,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:49,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:49,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:49,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:49,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:49,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:49,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:49,642 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,642 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:49,642 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@636eec98[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:49,644 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:49,645 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,646 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,646 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:49,646 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:49,646 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:49,646 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,646 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:49,647 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:49,647 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:49,647 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,647 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:49,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:49,649 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:49,649 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:49,649 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:49,649 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:49,650 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:49,652 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,652 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,652 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:49,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:49,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:49,653 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:49,655 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:49,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:49,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:49,657 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:49,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:49,656 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:49,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,657 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:49,657 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:49,657 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,657 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,657 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,657 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,657 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:49,657 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:49,665 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/dc73ee5beb814bf6a6a0930de2665206 2023-04-13 14:55:49,668 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/23c20183717b4979988eb569fe99e6f4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/23c20183717b4979988eb569fe99e6f4 2023-04-13 14:55:49,671 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/23c20183717b4979988eb569fe99e6f4, entries=200, sequenceid=346, filesize=14.3 K 2023-04-13 14:55:49,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/b9203e949afe47f5bbe7e3224f6d21b7 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/b9203e949afe47f5bbe7e3224f6d21b7 2023-04-13 14:55:49,676 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/b9203e949afe47f5bbe7e3224f6d21b7, entries=150, sequenceid=346, filesize=11.9 K 2023-04-13 14:55:49,677 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/dc73ee5beb814bf6a6a0930de2665206 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/dc73ee5beb814bf6a6a0930de2665206 2023-04-13 14:55:49,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/dc73ee5beb814bf6a6a0930de2665206, entries=150, sequenceid=346, filesize=11.9 K 2023-04-13 14:55:49,680 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~418.27 KB/428304, currentSize=33.54 KB/34350 for 46dced31cbc65aff91a95c238f6401f0 in 447ms, sequenceid=346, compaction requested=true 2023-04-13 14:55:49,680 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:49,680 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:49,680 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:55:49,680 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:55:49,680 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:49,680 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:49,682 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48704 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:55:49,682 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/B is initiating minor compaction (all files) 2023-04-13 14:55:49,682 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/B in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:49,682 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/0d5c2e91523e42fd80da88ac59f6a834, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/73486f2ddd484217b9aaf2d302a01edb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/c73d5550ae984c21964ee4f4d6ff5e44, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/b9203e949afe47f5bbe7e3224f6d21b7] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=47.6 K 2023-04-13 14:55:49,683 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 0d5c2e91523e42fd80da88ac59f6a834, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681397745469 2023-04-13 14:55:49,683 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 73486f2ddd484217b9aaf2d302a01edb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1681397746227 2023-04-13 14:55:49,683 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting c73d5550ae984c21964ee4f4d6ff5e44, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681397747380 2023-04-13 14:55:49,684 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53534 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:55:49,684 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting b9203e949afe47f5bbe7e3224f6d21b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1681397748880 2023-04-13 14:55:49,684 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/A is initiating minor compaction (all files) 2023-04-13 14:55:49,684 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/A in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:49,684 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/74081aff99d144aa81022acb6af20ffd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6ef4273160d843249bea6c72dc7f2913, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/15698f856129465f84ca792a067f2a45, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/23c20183717b4979988eb569fe99e6f4] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=52.3 K 2023-04-13 14:55:49,684 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 74081aff99d144aa81022acb6af20ffd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681397745469 2023-04-13 14:55:49,685 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 6ef4273160d843249bea6c72dc7f2913, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1681397746227 2023-04-13 14:55:49,685 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 15698f856129465f84ca792a067f2a45, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681397747380 2023-04-13 14:55:49,685 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 23c20183717b4979988eb569fe99e6f4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1681397748880 2023-04-13 14:55:49,691 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#B#compaction#422 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:49,695 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#A#compaction#423 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:49,757 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:49,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:49,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:49,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:49,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:49,760 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:49,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:49,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:49,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:49,761 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,761 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,769 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:49,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:49,769 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,770 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,770 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:49,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:49,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:49,771 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:49,771 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:49,771 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:49,771 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:49,771 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:49,771 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:49,771 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:49,771 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-13 14:55:49,772 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:49,772 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:49,772 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:49,772 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:49,772 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:49,772 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:49,779 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/314ce864cbaa4bf39953bd2e85dafe3c 2023-04-13 14:55:49,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:49,873 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:49,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681397749891, max timestamp=1681397749910Num uniques 859; 2023-04-13 14:55:49,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-13 14:55:49,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681397749891, max timestamp=1681397749910Num uniques 859; 2023-04-13 14:55:49,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681397749891, max timestamp=1681397749910Num uniques 859; 2023-04-13 14:55:49,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-13 14:55:49,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-13 14:55:49,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:49,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397809952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:49,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397809953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:49,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:49,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397809956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:50,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397810057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:50,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53296 deadline: 1681397810063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:50,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397810063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,106 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/344e4422efba4f6e99b1f81584d0bef8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/344e4422efba4f6e99b1f81584d0bef8 2023-04-13 14:55:50,110 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/B of 46dced31cbc65aff91a95c238f6401f0 into 344e4422efba4f6e99b1f81584d0bef8(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:50,110 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:50,110 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/B, priority=12, startTime=1681397749680; duration=0sec 2023-04-13 14:55:50,111 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:50,111 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:55:50,112 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48704 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:55:50,112 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/C is initiating minor compaction (all files) 2023-04-13 14:55:50,113 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/C in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:50,113 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/1c90f2f5cffe427f87dc531c0e17c3a8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5d5c75c4d9764384865107dbf47e35f2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/15a71c86b5474276a8277d2dd9b86175, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/dc73ee5beb814bf6a6a0930de2665206] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=47.6 K 2023-04-13 14:55:50,113 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 1c90f2f5cffe427f87dc531c0e17c3a8, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681397745469 2023-04-13 14:55:50,114 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 5d5c75c4d9764384865107dbf47e35f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1681397746227 2023-04-13 14:55:50,114 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 15a71c86b5474276a8277d2dd9b86175, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681397747380 2023-04-13 14:55:50,115 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting dc73ee5beb814bf6a6a0930de2665206, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1681397748880 2023-04-13 14:55:50,115 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/86086e409f7940109715e04ed67d4e1c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/86086e409f7940109715e04ed67d4e1c 2023-04-13 14:55:50,119 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/A of 46dced31cbc65aff91a95c238f6401f0 into 86086e409f7940109715e04ed67d4e1c(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:50,119 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:50,119 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/A, priority=12, startTime=1681397749680; duration=0sec 2023-04-13 14:55:50,120 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:50,122 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#C#compaction#426 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:50,129 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/abf56a2b3ce54954a4497e656c54c875 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/abf56a2b3ce54954a4497e656c54c875 2023-04-13 14:55:50,132 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/C of 46dced31cbc65aff91a95c238f6401f0 into abf56a2b3ce54954a4497e656c54c875(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:50,132 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:50,132 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/C, priority=12, startTime=1681397749680; duration=0sec 2023-04-13 14:55:50,132 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:50,190 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/aaf736ae904d4be0aecac655f475e990 2023-04-13 14:55:50,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:50,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397810266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:50,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397810270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:50,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397810270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:50,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397810573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:50,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397810573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:50,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397810573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,600 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/c9a38a37d0034bfb9dc1d944bc9114c6 2023-04-13 14:55:50,604 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/314ce864cbaa4bf39953bd2e85dafe3c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/314ce864cbaa4bf39953bd2e85dafe3c 2023-04-13 14:55:50,607 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/314ce864cbaa4bf39953bd2e85dafe3c, entries=150, sequenceid=354, filesize=11.9 K 2023-04-13 14:55:50,608 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/aaf736ae904d4be0aecac655f475e990 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/aaf736ae904d4be0aecac655f475e990 2023-04-13 14:55:50,612 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/aaf736ae904d4be0aecac655f475e990, entries=150, sequenceid=354, filesize=11.9 K 2023-04-13 14:55:50,613 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/c9a38a37d0034bfb9dc1d944bc9114c6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c9a38a37d0034bfb9dc1d944bc9114c6 2023-04-13 14:55:50,617 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c9a38a37d0034bfb9dc1d944bc9114c6, entries=150, sequenceid=354, filesize=11.9 K 2023-04-13 14:55:50,617 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 46dced31cbc65aff91a95c238f6401f0 in 846ms, sequenceid=354, compaction requested=false 2023-04-13 14:55:50,617 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:50,617 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:50,619 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:50,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:50,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ffffeae[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:50,621 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:50,621 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:50,622 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:50,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:50,623 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,623 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:50,624 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:50,624 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,624 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:50,624 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:50,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:50,626 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:50,627 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:50,627 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:50,627 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:50,627 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:50,629 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:50,629 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:50,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:50,629 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:50,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:50,632 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,634 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,640 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:50,640 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:50,640 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:50,640 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:50,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:50,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:50,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:50,640 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:50,640 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:50,640 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:50,640 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:50,640 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:50,640 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:50,641 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:50,641 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:50,740 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:50,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:50,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:50,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:50,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:50,743 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:50,743 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:50,744 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:50,744 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:50,744 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:50,744 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:50,745 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,745 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:50,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:50,745 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:50,746 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:50,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:50,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:50,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:50,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:50,747 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:50,747 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:50,747 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:50,747 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-04-13 14:55:50,747 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:50,747 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:50,747 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:50,747 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:50,747 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:50,747 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:51,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:51,083 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:51,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:51,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397811088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:51,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397811092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:51,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397811092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,170 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/124132879274472d8e1a95b140d95154 2023-04-13 14:55:51,190 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/5e6e842785d54559975633a4734aaadc 2023-04-13 14:55:51,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:51,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397811195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:51,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397811199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:51,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397811199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:51,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397811401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:51,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397811405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:51,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397811407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,604 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/9df4170b7dcb423ba1b0a7867ca621ee 2023-04-13 14:55:51,608 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/124132879274472d8e1a95b140d95154 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/124132879274472d8e1a95b140d95154 2023-04-13 14:55:51,612 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/124132879274472d8e1a95b140d95154, entries=150, sequenceid=394, filesize=11.9 K 2023-04-13 14:55:51,613 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/5e6e842785d54559975633a4734aaadc as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/5e6e842785d54559975633a4734aaadc 2023-04-13 14:55:51,616 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/5e6e842785d54559975633a4734aaadc, entries=150, sequenceid=394, filesize=11.9 K 2023-04-13 14:55:51,617 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/9df4170b7dcb423ba1b0a7867ca621ee as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/9df4170b7dcb423ba1b0a7867ca621ee 2023-04-13 14:55:51,620 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/9df4170b7dcb423ba1b0a7867ca621ee, entries=150, sequenceid=394, filesize=11.9 K 2023-04-13 14:55:51,621 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=33.54 KB/34350 for 46dced31cbc65aff91a95c238f6401f0 in 874ms, sequenceid=394, compaction requested=true 2023-04-13 14:55:51,621 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:51,621 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:51,622 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:51,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:51,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,623 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cf9290d[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:51,624 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:51,624 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,626 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:51,626 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,626 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:51,627 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,627 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:51,627 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:51,627 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:51,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:51,630 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:51,630 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:51,630 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:51,630 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:51,630 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:51,631 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:51,631 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,631 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:51,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:51,632 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,633 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,635 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:51,635 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:51,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:51,635 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:51,635 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:51,635 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:51,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:51,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:51,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:51,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,636 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:51,636 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:51,636 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:51,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:51,714 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:55:51,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:51,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:51,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:51,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:51,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:51,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:51,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/c04d20c5beaf406b9254e52de81de04e 2023-04-13 14:55:51,744 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/4689054a16f946f4972142e1be69b583 2023-04-13 14:55:51,748 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:51,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:51,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:51,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:51,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:51,755 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:51,755 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:51,755 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:51,755 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:51,756 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,756 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:51,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:51,759 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:51,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:51,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:51,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:51,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:51,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:51,761 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:51,761 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:51,761 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/83203eb5194b4be78a8868b20eeb47dd 2023-04-13 14:55:51,761 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:51,761 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:51,761 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:51,761 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:51,761 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:51,762 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:51,762 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:51,762 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:51,762 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:51,762 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,762 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:51,762 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:51,763 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:51,764 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:51,764 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,765 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/c04d20c5beaf406b9254e52de81de04e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/c04d20c5beaf406b9254e52de81de04e 2023-04-13 14:55:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:51,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:51,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6e9724[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:51,766 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:51,766 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,767 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:51,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:51,769 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/c04d20c5beaf406b9254e52de81de04e, entries=150, sequenceid=404, filesize=11.9 K 2023-04-13 14:55:51,769 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:51,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/4689054a16f946f4972142e1be69b583 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/4689054a16f946f4972142e1be69b583 2023-04-13 14:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:51,771 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:51,772 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:51,772 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:51,772 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:51,772 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:51,774 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,774 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:51,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,774 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:51,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:51,775 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,776 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,778 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:51,779 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:51,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:51,779 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,779 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:51,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:51,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:51,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:51,779 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:51,779 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:51,779 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,779 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,779 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,779 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:51,779 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:51,779 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/4689054a16f946f4972142e1be69b583, entries=100, sequenceid=404, filesize=9.5 K 2023-04-13 14:55:51,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/83203eb5194b4be78a8868b20eeb47dd as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/83203eb5194b4be78a8868b20eeb47dd 2023-04-13 14:55:51,786 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/83203eb5194b4be78a8868b20eeb47dd, entries=100, sequenceid=404, filesize=9.5 K 2023-04-13 14:55:51,787 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=114.05 KB/116790 for 46dced31cbc65aff91a95c238f6401f0 in 73ms, sequenceid=404, compaction requested=true 2023-04-13 14:55:51,787 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:51,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:51,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:51,787 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:55:51,787 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:55:51,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:51,788 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46846 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:55:51,788 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/B is initiating minor compaction (all files) 2023-04-13 14:55:51,788 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/B in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:51,789 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/344e4422efba4f6e99b1f81584d0bef8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/aaf736ae904d4be0aecac655f475e990, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/5e6e842785d54559975633a4734aaadc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/4689054a16f946f4972142e1be69b583] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=45.7 K 2023-04-13 14:55:51,789 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 344e4422efba4f6e99b1f81584d0bef8, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1681397748880 2023-04-13 14:55:51,789 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49286 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:55:51,789 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/A is initiating minor compaction (all files) 2023-04-13 14:55:51,789 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/A in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:51,789 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting aaf736ae904d4be0aecac655f475e990, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1681397749238 2023-04-13 14:55:51,789 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/86086e409f7940109715e04ed67d4e1c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/314ce864cbaa4bf39953bd2e85dafe3c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/124132879274472d8e1a95b140d95154, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/c04d20c5beaf406b9254e52de81de04e] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=48.1 K 2023-04-13 14:55:51,790 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 5e6e842785d54559975633a4734aaadc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1681397749951 2023-04-13 14:55:51,790 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 86086e409f7940109715e04ed67d4e1c, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1681397748880 2023-04-13 14:55:51,790 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 4689054a16f946f4972142e1be69b583, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681397751709 2023-04-13 14:55:51,790 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 314ce864cbaa4bf39953bd2e85dafe3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1681397749238 2023-04-13 14:55:51,791 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 124132879274472d8e1a95b140d95154, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1681397749951 2023-04-13 14:55:51,791 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting c04d20c5beaf406b9254e52de81de04e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681397751090 2023-04-13 14:55:51,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:51,792 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=127.47 KB heapSize=379.97 KB 2023-04-13 14:55:51,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397751763, max timestamp=1681397751791Num uniques 859; 2023-04-13 14:55:51,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:51,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:51,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397751763, max timestamp=1681397751791Num uniques 859; 2023-04-13 14:55:51,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:51,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:51,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:51,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:51,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:51,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:51,801 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#B#compaction#435 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:51,802 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#A#compaction#436 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:51,834 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/6e306cd254a94b1b9661a560a6b9efa8 2023-04-13 14:55:51,839 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/372adc5de5444750959402d2bafd9ffb as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/372adc5de5444750959402d2bafd9ffb 2023-04-13 14:55:51,846 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/B of 46dced31cbc65aff91a95c238f6401f0 into 372adc5de5444750959402d2bafd9ffb(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:51,846 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:51,846 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/B, priority=12, startTime=1681397751787; duration=0sec 2023-04-13 14:55:51,846 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:51,846 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:55:51,847 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46846 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:55:51,847 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/C is initiating minor compaction (all files) 2023-04-13 14:55:51,847 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/C in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:51,847 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/abf56a2b3ce54954a4497e656c54c875, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c9a38a37d0034bfb9dc1d944bc9114c6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/9df4170b7dcb423ba1b0a7867ca621ee, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/83203eb5194b4be78a8868b20eeb47dd] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=45.7 K 2023-04-13 14:55:51,848 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting abf56a2b3ce54954a4497e656c54c875, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1681397748880 2023-04-13 14:55:51,848 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting c9a38a37d0034bfb9dc1d944bc9114c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1681397749238 2023-04-13 14:55:51,848 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 9df4170b7dcb423ba1b0a7867ca621ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1681397749951 2023-04-13 14:55:51,849 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 83203eb5194b4be78a8868b20eeb47dd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681397751709 2023-04-13 14:55:51,858 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#C#compaction#437 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:51,858 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/90cf1691fd49453793cb5a7884a70b3c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/90cf1691fd49453793cb5a7884a70b3c 2023-04-13 14:55:51,874 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/A of 46dced31cbc65aff91a95c238f6401f0 into 90cf1691fd49453793cb5a7884a70b3c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:51,874 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:51,874 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/A, priority=12, startTime=1681397751787; duration=0sec 2023-04-13 14:55:51,874 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:51,875 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/7469c1ee554d404eaba2844e7dde0199 2023-04-13 14:55:51,879 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:51,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:51,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:51,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:51,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:51,884 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:51,884 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:51,884 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:51,884 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:51,885 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,885 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:51,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:51,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397811855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,886 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:51,886 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:51,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:51,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397811855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:51,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397811855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,886 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,887 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:51,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:51,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:51,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:51,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:51,896 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:51,896 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:51,896 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:51,898 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:51,898 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:51,898 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:51,898 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:51,898 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:51,898 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:51,898 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:51,898 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,898 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:51,898 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:51,900 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,900 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:51,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:51,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e70acdd[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:51,902 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:51,902 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,903 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/ff1e9124f28040da8000ede27793c067 2023-04-13 14:55:51,904 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,904 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,904 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:51,906 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:51,906 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:51,906 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,906 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:51,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/6e306cd254a94b1b9661a560a6b9efa8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6e306cd254a94b1b9661a560a6b9efa8 2023-04-13 14:55:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:51,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:51,908 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:51,908 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:51,908 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:51,909 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:51,909 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:51,910 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:51,910 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,911 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,910 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:51,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6e306cd254a94b1b9661a560a6b9efa8, entries=200, sequenceid=427, filesize=14.3 K 2023-04-13 14:55:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,912 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:51,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/7469c1ee554d404eaba2844e7dde0199 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7469c1ee554d404eaba2844e7dde0199 2023-04-13 14:55:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,917 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,917 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:51,917 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:51,917 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:51,917 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:51,917 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:51,917 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,917 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:51,917 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:51,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:51,917 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:51,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:51,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:51,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:51,918 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:51,918 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:51,919 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7469c1ee554d404eaba2844e7dde0199, entries=150, sequenceid=427, filesize=11.9 K 2023-04-13 14:55:51,920 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/ff1e9124f28040da8000ede27793c067 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ff1e9124f28040da8000ede27793c067 2023-04-13 14:55:51,925 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ff1e9124f28040da8000ede27793c067, entries=150, sequenceid=427, filesize=11.9 K 2023-04-13 14:55:51,926 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~71.56 KB/73280, heapSize ~203.97 KB/208864, currentSize=107.34 KB/109920 for 46dced31cbc65aff91a95c238f6401f0 in 134ms, sequenceid=427, compaction requested=false 2023-04-13 14:55:51,926 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:51,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:51,990 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 14:55:51,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397751852, max timestamp=1681397751990Num uniques 859; 2023-04-13 14:55:51,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:51,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397751852, max timestamp=1681397751990Num uniques 859; 2023-04-13 14:55:51,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397751852, max timestamp=1681397751990Num uniques 859; 2023-04-13 14:55:51,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:51,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:51,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:51,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:51,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:51,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:51,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:51,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:52,001 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/3a0add52ee1746c09283d802d3715943 2023-04-13 14:55:52,010 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/450c1bedd67a400d9cd301f9bb97e1a4 2023-04-13 14:55:52,018 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:52,018 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/ff48d9e4bf984ec1b24573da0eb95f4b 2023-04-13 14:55:52,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:52,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:52,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:52,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:52,022 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:52,022 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,022 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:52,022 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:52,022 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,022 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,023 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/3a0add52ee1746c09283d802d3715943 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3a0add52ee1746c09283d802d3715943 2023-04-13 14:55:52,024 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,024 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,025 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,025 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:52,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:52,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:52,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:52,025 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:52,026 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,026 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:52,026 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:52,026 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:52,026 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,026 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:52,026 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:52,026 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,026 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:52,026 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:52,026 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,026 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:52,026 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3a0add52ee1746c09283d802d3715943, entries=250, sequenceid=450, filesize=16.7 K 2023-04-13 14:55:52,026 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:52,027 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/450c1bedd67a400d9cd301f9bb97e1a4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/450c1bedd67a400d9cd301f9bb97e1a4 2023-04-13 14:55:52,030 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,030 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,031 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,031 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,036 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/450c1bedd67a400d9cd301f9bb97e1a4, entries=150, sequenceid=450, filesize=11.9 K 2023-04-13 14:55:52,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:52,037 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:52,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e1a3733[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:52,037 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,037 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/ff48d9e4bf984ec1b24573da0eb95f4b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ff48d9e4bf984ec1b24573da0eb95f4b 2023-04-13 14:55:52,041 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ff48d9e4bf984ec1b24573da0eb95f4b, entries=150, sequenceid=450, filesize=11.9 K 2023-04-13 14:55:52,044 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~81.42 KB/83376, currentSize=107.34 KB/109920 for 46dced31cbc65aff91a95c238f6401f0 in 54ms, sequenceid=450, compaction requested=true 2023-04-13 14:55:52,044 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:52,044 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:52,044 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 14:55:52,044 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-13 14:55:52,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:52,044 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=114.05 KB heapSize=340.13 KB 2023-04-13 14:55:52,045 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-04-13 14:55:52,045 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:52,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681397752019, max timestamp=1681397752040Num uniques 859; 2023-04-13 14:55:52,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681397752019, max timestamp=1681397752040Num uniques 859; 2023-04-13 14:55:52,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:52,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:52,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681397752019, max timestamp=1681397752040Num uniques 859; 2023-04-13 14:55:52,046 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,046 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:52,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:52,046 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 14:55:52,046 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 14:55:52,046 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:52,046 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. because compaction request was cancelled 2023-04-13 14:55:52,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:52,046 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:52,046 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:52,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:52,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:52,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,047 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,047 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44560 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:52,048 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/A is initiating minor compaction (all files) 2023-04-13 14:55:52,048 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/A in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,048 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/90cf1691fd49453793cb5a7884a70b3c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6e306cd254a94b1b9661a560a6b9efa8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3a0add52ee1746c09283d802d3715943] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=43.5 K 2023-04-13 14:55:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,048 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 90cf1691fd49453793cb5a7884a70b3c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681397749955 2023-04-13 14:55:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,049 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 6e306cd254a94b1b9661a560a6b9efa8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681397751778 2023-04-13 14:55:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,049 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 3a0add52ee1746c09283d802d3715943, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681397751852 2023-04-13 14:55:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:52,061 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:52,061 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:52,061 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:52,061 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:52,061 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:52,064 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,064 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,064 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,066 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,066 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#A#compaction#444 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,073 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/1579ac04717345aa801b9cf95018b41f 2023-04-13 14:55:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,073 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,076 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,076 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,077 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,077 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,077 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,077 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,077 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,077 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,077 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:52,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:52,077 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,078 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:52,078 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:52,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:52,084 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/8c276456052f4dd997913e95cc8ffc00 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8c276456052f4dd997913e95cc8ffc00 2023-04-13 14:55:52,088 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/A of 46dced31cbc65aff91a95c238f6401f0 into 8c276456052f4dd997913e95cc8ffc00(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:52,088 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:52,088 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/A, priority=13, startTime=1681397752044; duration=0sec 2023-04-13 14:55:52,088 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:52,089 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:52,089 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:52,089 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/B is initiating minor compaction (all files) 2023-04-13 14:55:52,089 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/B in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,090 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/372adc5de5444750959402d2bafd9ffb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7469c1ee554d404eaba2844e7dde0199, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/450c1bedd67a400d9cd301f9bb97e1a4] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=36.4 K 2023-04-13 14:55:52,090 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 372adc5de5444750959402d2bafd9ffb, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681397749955 2023-04-13 14:55:52,090 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 7469c1ee554d404eaba2844e7dde0199, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681397751778 2023-04-13 14:55:52,091 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 450c1bedd67a400d9cd301f9bb97e1a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681397751853 2023-04-13 14:55:52,102 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#B#compaction#446 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:52,112 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/31e70373df554d399a0d41047e54ceab as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/31e70373df554d399a0d41047e54ceab 2023-04-13 14:55:52,116 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/B of 46dced31cbc65aff91a95c238f6401f0 into 31e70373df554d399a0d41047e54ceab(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:52,116 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:52,116 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/B, priority=13, startTime=1681397752044; duration=0sec 2023-04-13 14:55:52,116 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:52,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397752105, max timestamp=1681397752119Num uniques 859; 2023-04-13 14:55:52,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:55:52,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397752105, max timestamp=1681397752119Num uniques 859; 2023-04-13 14:55:52,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:52,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397752105, max timestamp=1681397752119Num uniques 859; 2023-04-13 14:55:52,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:52,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681397752150, max timestamp=1681397752164Num uniques 1068; 2023-04-13 14:55:52,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681397752150, max timestamp=1681397752164Num uniques 1018; 2023-04-13 14:55:52,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-13 14:55:52,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-13 14:55:52,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681397752150, max timestamp=1681397752164Num uniques 1018; 2023-04-13 14:55:52,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-13 14:55:52,178 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:52,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:52,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:52,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:52,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:52,183 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:52,183 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,183 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:52,183 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:52,183 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,183 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,184 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,184 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,185 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,185 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:52,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:52,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:52,187 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,187 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:52,187 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:52,187 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:52,189 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,189 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,189 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,189 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:52,193 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:52,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1baed4bc[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:52,193 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,195 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,195 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:52,195 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,195 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:52,195 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:52,195 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:52,197 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:52,197 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,197 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,197 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:52,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:52,199 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:52,199 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:52,199 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:52,199 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:52,199 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:52,201 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,201 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,201 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,202 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,203 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,206 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,206 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,206 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:52,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:52,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,206 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:52,206 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:52,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:52,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397812201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681397752196, max timestamp=1681397752208Num uniques 1127; 2023-04-13 14:55:52,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-13 14:55:52,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681397752196, max timestamp=1681397752208Num uniques 1127; 2023-04-13 14:55:52,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681397752196, max timestamp=1681397752208Num uniques 1127; 2023-04-13 14:55:52,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-13 14:55:52,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-13 14:55:52,277 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/c0470cadaea943d6a66a01eb908d101f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c0470cadaea943d6a66a01eb908d101f 2023-04-13 14:55:52,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681397752250, max timestamp=1681397752277Num uniques 1086; 2023-04-13 14:55:52,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681397752250, max timestamp=1681397752277Num uniques 1086; 2023-04-13 14:55:52,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-13 14:55:52,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-13 14:55:52,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681397752250, max timestamp=1681397752277Num uniques 1086; 2023-04-13 14:55:52,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-13 14:55:52,283 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/C of 46dced31cbc65aff91a95c238f6401f0 into c0470cadaea943d6a66a01eb908d101f(size=12.6 K), total size for store is 36.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:52,283 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:52,283 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/C, priority=12, startTime=1681397751787; duration=0sec 2023-04-13 14:55:52,283 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:52,306 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:52,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:52,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:52,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:52,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:52,316 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:52,316 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,316 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:52,316 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:52,316 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,316 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,323 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,323 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,324 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,324 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:52,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:52,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:52,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:52,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:52,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:52,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:52,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:52,325 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,325 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:52,326 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:52,326 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,326 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:52,326 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:52,326 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,326 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:52,326 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:52,327 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,327 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,328 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,328 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:52,329 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:52,329 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e7a75a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:52,330 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,330 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,330 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:52,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:52,331 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:52,332 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,333 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:52,333 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,333 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:52,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:52,335 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:52,335 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:52,335 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:52,335 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:52,335 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:52,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,338 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,340 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,342 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,342 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,342 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:52,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:52,342 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,342 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,342 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,342 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,342 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,342 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,342 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,342 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:52,342 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:52,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:52,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681397752330, max timestamp=1681397752345Num uniques 1045; 2023-04-13 14:55:52,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-13 14:55:52,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681397752330, max timestamp=1681397752345Num uniques 1045; 2023-04-13 14:55:52,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-13 14:55:52,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681397752330, max timestamp=1681397752345Num uniques 1045; 2023-04-13 14:55:52,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-13 14:55:52,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681397752395, max timestamp=1681397752403Num uniques 1104; 2023-04-13 14:55:52,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-13 14:55:52,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681397752395, max timestamp=1681397752403Num uniques 1104; 2023-04-13 14:55:52,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-13 14:55:52,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681397752395, max timestamp=1681397752403Num uniques 1104; 2023-04-13 14:55:52,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-13 14:55:52,443 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:52,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:52,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:52,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:52,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:52,445 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:52,446 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,446 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:52,446 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:52,446 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,446 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,447 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,447 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,447 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,448 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:52,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:52,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:52,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:52,450 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:52,450 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,450 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:52,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9745, totalHeapSize=22392, min timestamp=1681397752433, max timestamp=1681397752449Num uniques 1063; 2023-04-13 14:55:52,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48675, new segment data size=9745, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-13 14:55:52,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9745, totalHeapSize=22392, min timestamp=1681397752433, max timestamp=1681397752449Num uniques 1063; 2023-04-13 14:55:52,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9745, totalHeapSize=22392, min timestamp=1681397752433, max timestamp=1681397752449Num uniques 1063; 2023-04-13 14:55:52,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48675, new segment data size=9745, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-13 14:55:52,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:52,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:52,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48675, new segment data size=9745, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-13 14:55:52,452 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,452 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:52,452 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:52,452 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:52,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:52,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:52,452 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:52,453 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,453 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,454 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,454 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e8af57f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:52,455 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:52,455 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,458 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,458 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,458 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:52,459 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,459 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:52,459 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,460 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:52,462 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:52,462 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:52,462 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:52,462 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:52,462 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:52,464 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,464 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,464 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,465 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,466 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,467 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,467 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,468 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,468 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,468 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:52,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:52,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,467 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:52,468 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:52,468 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,468 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,468 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,468 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,468 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:52,499 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/14fa4665da8c42a6ac887e42a591fc88 2023-04-13 14:55:52,508 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/ec3189d2c1574361a2c7b9553b39aeef 2023-04-13 14:55:52,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681397752480, max timestamp=1681397752509Num uniques 1072; 2023-04-13 14:55:52,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681397752480, max timestamp=1681397752509Num uniques 1072; 2023-04-13 14:55:52,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-13 14:55:52,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-13 14:55:52,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681397752480, max timestamp=1681397752509Num uniques 1072; 2023-04-13 14:55:52,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-13 14:55:52,513 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/1579ac04717345aa801b9cf95018b41f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/1579ac04717345aa801b9cf95018b41f 2023-04-13 14:55:52,517 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/1579ac04717345aa801b9cf95018b41f, entries=200, sequenceid=471, filesize=14.3 K 2023-04-13 14:55:52,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/14fa4665da8c42a6ac887e42a591fc88 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/14fa4665da8c42a6ac887e42a591fc88 2023-04-13 14:55:52,524 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/14fa4665da8c42a6ac887e42a591fc88, entries=150, sequenceid=471, filesize=11.9 K 2023-04-13 14:55:52,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/ec3189d2c1574361a2c7b9553b39aeef as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ec3189d2c1574361a2c7b9553b39aeef 2023-04-13 14:55:52,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ec3189d2c1574361a2c7b9553b39aeef, entries=150, sequenceid=471, filesize=11.9 K 2023-04-13 14:55:52,530 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~81.42 KB/83376, currentSize=67.09 KB/68700 for 46dced31cbc65aff91a95c238f6401f0 in 486ms, sequenceid=471, compaction requested=true 2023-04-13 14:55:52,530 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:52,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:52,530 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-13 14:55:52,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:52,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:52,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 14:55:52,530 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=73.80 KB heapSize=204.47 KB 2023-04-13 14:55:52,530 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:55:52,531 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 14:55:52,531 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 14:55:52,531 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. because compaction request was cancelled 2023-04-13 14:55:52,531 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-13 14:55:52,534 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:52,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:52,535 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:52,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:52,535 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:52,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:52,535 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 14:55:52,535 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:55:52,535 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 14:55:52,535 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/C is initiating minor compaction (all files) 2023-04-13 14:55:52,535 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. because compaction request was cancelled 2023-04-13 14:55:52,535 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/C in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,535 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c0470cadaea943d6a66a01eb908d101f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ff1e9124f28040da8000ede27793c067, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ff48d9e4bf984ec1b24573da0eb95f4b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ec3189d2c1574361a2c7b9553b39aeef] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=48.3 K 2023-04-13 14:55:52,535 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting c0470cadaea943d6a66a01eb908d101f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681397749955 2023-04-13 14:55:52,536 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting ff1e9124f28040da8000ede27793c067, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681397751778 2023-04-13 14:55:52,536 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting ff48d9e4bf984ec1b24573da0eb95f4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681397751853 2023-04-13 14:55:52,537 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting ec3189d2c1574361a2c7b9553b39aeef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681397752019 2023-04-13 14:55:52,550 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#C#compaction#449 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:52,568 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:52,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:52,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:52,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:52,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:52,571 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:52,571 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,571 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:52,571 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:52,572 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,572 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,572 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/f3a1536d2aac415faa8f5ef851a2cfaf as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/f3a1536d2aac415faa8f5ef851a2cfaf 2023-04-13 14:55:52,574 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,574 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,574 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,574 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:52,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:52,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:52,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:52,575 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:52,575 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,575 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:52,575 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:52,575 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:52,575 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,575 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:52,576 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:52,576 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:52,576 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/C of 46dced31cbc65aff91a95c238f6401f0 into f3a1536d2aac415faa8f5ef851a2cfaf(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,576 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:52,576 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/C, priority=12, startTime=1681397752530; duration=0sec 2023-04-13 14:55:52,576 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:52,578 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,578 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,578 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,578 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:52,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3166f632[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:52,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:52,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,581 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,581 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:52,581 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:52,581 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:52,581 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,581 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,582 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:52,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:52,584 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:52,584 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:52,584 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:52,584 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:52,584 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:52,587 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,587 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,587 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,589 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,590 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,591 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,591 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,591 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,591 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:52,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:52,591 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,592 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,592 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,592 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,592 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,592 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:52,592 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:52,592 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:52,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397812580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,600 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:52,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397812593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:52,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397812594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,692 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:52,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:52,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:52,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:52,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:52,695 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:52,695 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,696 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:52,696 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:52,696 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,696 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,697 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,697 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,697 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,698 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:52,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:52,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:52,699 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:52,699 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,699 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:52,699 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:52,699 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:52,699 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,699 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:52,699 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:52,699 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,699 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:52,699 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:52,699 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,699 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:52,700 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:52,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:52,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397812695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,702 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,703 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,703 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,703 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:52,704 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:52,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c80cf96[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:52,704 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:52,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:52,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:52,706 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,706 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:52,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:52,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397812702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:52,708 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397812702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,708 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:52,708 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,708 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:52,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:52,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:52,711 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:52,711 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:52,711 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:52,711 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:52,711 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:52,712 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,712 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,712 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,713 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,714 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:52,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:52,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,717 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,717 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,717 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,717 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,717 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,717 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,716 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:52,718 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:52,718 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:52,818 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:52,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:52,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:52,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:52,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:52,821 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:52,822 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,822 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:52,822 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:52,822 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,822 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,824 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,824 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,824 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,824 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:52,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:52,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:52,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:52,826 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:52,826 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,826 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:52,826 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:52,826 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:52,829 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,829 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:52,829 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:52,829 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,829 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:52,829 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:52,829 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,829 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:52,829 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:52,830 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,830 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:52,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64657e2b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:52,832 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:52,832 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,833 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,834 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:52,834 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:52,834 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:52,834 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,834 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:52,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:52,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,835 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:52,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:52,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:52,838 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:52,838 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:52,838 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:52,838 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:52,838 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:52,845 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,845 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,845 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,847 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,849 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:52,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:52,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:52,849 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,854 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:52,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,854 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:52,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:52,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397812901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:52,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397812909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:52,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397812910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,955 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:52,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:52,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:52,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:52,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:52,959 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:52,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:52,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:52,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,961 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,961 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,961 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,961 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:52,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:52,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:52,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:52,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:52,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:52,963 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:52,965 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,965 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/4fb28703c9c145c9bd48f6fdf3a689a4 2023-04-13 14:55:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:52,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d6d4ed5[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:52,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:52,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,975 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,975 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:52,975 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:52,975 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:52,977 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:52,977 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,977 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:52,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:52,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:52,977 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:52,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:52,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:52,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:52,977 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:52,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,979 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:52,980 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:52,981 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:52,986 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:52,986 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:52,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:52,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:52,986 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,986 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,986 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:52,986 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:52,986 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,986 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:52,986 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:52,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:52,986 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:52,986 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:52,987 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:52,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:52,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:52,987 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/593d619df2bb4af18026c7695cdcbb37 2023-04-13 14:55:53,086 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:53,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:53,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:53,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:53,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:53,090 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:53,090 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:53,090 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:53,090 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:53,090 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,090 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,093 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:53,093 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:53,093 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,093 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:53,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:53,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:53,094 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:53,094 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:53,094 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:53,094 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:53,094 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:53,095 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:53,095 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:53,095 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:53,095 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:53,096 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,096 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:53,097 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,097 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:53,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:53,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bbce62c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:53,100 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:53,100 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,101 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,101 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:53,101 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,101 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:53,101 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:53,101 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:53,102 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:53,102 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:53,102 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,102 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:53,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:53,105 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:53,105 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:53,105 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:53,105 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:53,105 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:53,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:53,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,110 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:53,110 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,112 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,115 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:53,115 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:53,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:53,115 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,115 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,115 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:53,115 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:53,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:53,116 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,116 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,116 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:53,116 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:53,116 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:53,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:53,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:53,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397813206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,216 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:53,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:53,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:53,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:53,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397813213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:53,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:53,219 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:53,219 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:53,219 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:53,219 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:53,219 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,220 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,221 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:53,221 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:53,221 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,221 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:53,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:53,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:53,222 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:53,222 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:53,223 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:53,223 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:53,223 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:53,223 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:53,223 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,223 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:53,223 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:53,224 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,224 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:53,224 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,224 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397813216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:53,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:53,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:53,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@119a7cb2[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:53,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,229 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:53,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:53,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:53,229 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,229 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:53,230 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,230 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:53,230 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:53,230 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:53,232 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:53,232 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:53,233 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:53,233 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:53,233 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:53,234 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:53,234 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,234 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:53,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:53,234 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,236 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,241 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:53,241 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:53,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:53,241 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:53,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:53,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:53,241 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,248 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,248 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,248 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:53,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:53,248 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,248 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,248 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,248 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:53,248 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:53,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:53,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53256 deadline: 1681397813281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,290 DEBUG [Thread-2924] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8204 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:55:53,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:53,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53246 deadline: 1681397813300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,309 DEBUG [Thread-2916] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8220 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:55:53,348 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:53,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:53,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:53,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:53,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:53,352 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:53,352 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:53,352 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:53,352 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:53,352 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,352 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,353 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:53,353 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:53,354 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,354 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:53,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:53,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:53,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:53,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:53,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:53,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:53,355 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:53,355 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:53,355 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:53,356 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:53,356 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,357 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:53,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:53,359 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:53,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ec1cc8c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:53,359 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,360 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:53,361 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,361 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:53,363 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:53,363 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:53,363 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:53,364 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:53,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:53,366 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:53,366 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:53,366 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:53,366 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:53,366 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:53,367 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,367 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:53,367 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:53,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:53,368 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:53,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:53,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:53,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:53,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,372 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:53,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:53,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:53,372 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:53,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:53,373 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:53,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:53,373 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:53,406 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/566509147a044f17ae2e73610c32811e 2023-04-13 14:55:53,410 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/4fb28703c9c145c9bd48f6fdf3a689a4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/4fb28703c9c145c9bd48f6fdf3a689a4 2023-04-13 14:55:53,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/4fb28703c9c145c9bd48f6fdf3a689a4, entries=350, sequenceid=624, filesize=21.4 K 2023-04-13 14:55:53,414 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/593d619df2bb4af18026c7695cdcbb37 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/593d619df2bb4af18026c7695cdcbb37 2023-04-13 14:55:53,417 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/593d619df2bb4af18026c7695cdcbb37, entries=150, sequenceid=624, filesize=11.9 K 2023-04-13 14:55:53,418 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/566509147a044f17ae2e73610c32811e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/566509147a044f17ae2e73610c32811e 2023-04-13 14:55:53,421 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/566509147a044f17ae2e73610c32811e, entries=150, sequenceid=624, filesize=11.9 K 2023-04-13 14:55:53,421 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~244.27 KB/250128, currentSize=93.93 KB/96180 for 46dced31cbc65aff91a95c238f6401f0 in 891ms, sequenceid=624, compaction requested=true 2023-04-13 14:55:53,421 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:53,422 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:53,422 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:53,422 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:53,422 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:53,422 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:53,423 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 49546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:53,423 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/A is initiating minor compaction (all files) 2023-04-13 14:55:53,423 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:53,423 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/A in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:53,423 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/B is initiating minor compaction (all files) 2023-04-13 14:55:53,423 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8c276456052f4dd997913e95cc8ffc00, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/1579ac04717345aa801b9cf95018b41f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/4fb28703c9c145c9bd48f6fdf3a689a4] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=48.4 K 2023-04-13 14:55:53,423 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/B in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:53,423 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/31e70373df554d399a0d41047e54ceab, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/14fa4665da8c42a6ac887e42a591fc88, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/593d619df2bb4af18026c7695cdcbb37] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=36.5 K 2023-04-13 14:55:53,423 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 8c276456052f4dd997913e95cc8ffc00, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681397751853 2023-04-13 14:55:53,423 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 31e70373df554d399a0d41047e54ceab, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681397751853 2023-04-13 14:55:53,424 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 1579ac04717345aa801b9cf95018b41f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681397752019 2023-04-13 14:55:53,424 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 14fa4665da8c42a6ac887e42a591fc88, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681397752019 2023-04-13 14:55:53,424 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 4fb28703c9c145c9bd48f6fdf3a689a4, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1681397752510 2023-04-13 14:55:53,424 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 593d619df2bb4af18026c7695cdcbb37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1681397752510 2023-04-13 14:55:53,431 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#B#compaction#452 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:53,434 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#A#compaction#453 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:53,461 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/26f60155853a413bbbd1c3619b35408c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/26f60155853a413bbbd1c3619b35408c 2023-04-13 14:55:53,464 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/491351f3b8dd421c83b5b56f2cfdc754 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/491351f3b8dd421c83b5b56f2cfdc754 2023-04-13 14:55:53,466 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/B of 46dced31cbc65aff91a95c238f6401f0 into 26f60155853a413bbbd1c3619b35408c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:53,466 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:53,466 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/B, priority=13, startTime=1681397753422; duration=0sec 2023-04-13 14:55:53,466 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:53,467 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-13 14:55:53,468 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 14:55:53,468 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 14:55:53,468 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. because compaction request was cancelled 2023-04-13 14:55:53,470 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/A of 46dced31cbc65aff91a95c238f6401f0 into 491351f3b8dd421c83b5b56f2cfdc754(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:53,470 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:53,470 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/A, priority=13, startTime=1681397753422; duration=0sec 2023-04-13 14:55:53,470 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:53,473 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:53,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:53,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:53,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:53,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:53,476 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:53,476 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:53,476 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:53,476 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:53,476 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,476 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,477 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,478 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:53,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:53,478 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,478 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:53,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:53,479 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:53,479 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:53,479 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 14:55:53,479 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:53,480 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:53,480 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:53,480 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:53,480 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:53,480 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:53,504 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/5e6d0757994243c39fcc5b88611e95c6 2023-04-13 14:55:53,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:53,719 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:53,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:53,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397813758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:53,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397813758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:53,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397813765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:53,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397813866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:53,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397813866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:53,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397813869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,914 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/6bff03f298474964ae76f75f96c71d47 2023-04-13 14:55:53,923 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/d7426f1057dd47ec8c442e5604dcaf68 2023-04-13 14:55:53,926 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/5e6d0757994243c39fcc5b88611e95c6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5e6d0757994243c39fcc5b88611e95c6 2023-04-13 14:55:53,931 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5e6d0757994243c39fcc5b88611e95c6, entries=150, sequenceid=644, filesize=11.9 K 2023-04-13 14:55:53,932 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/6bff03f298474964ae76f75f96c71d47 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6bff03f298474964ae76f75f96c71d47 2023-04-13 14:55:53,935 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6bff03f298474964ae76f75f96c71d47, entries=150, sequenceid=644, filesize=11.9 K 2023-04-13 14:55:53,936 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/d7426f1057dd47ec8c442e5604dcaf68 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/d7426f1057dd47ec8c442e5604dcaf68 2023-04-13 14:55:53,942 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/d7426f1057dd47ec8c442e5604dcaf68, entries=150, sequenceid=644, filesize=11.9 K 2023-04-13 14:55:53,943 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 46dced31cbc65aff91a95c238f6401f0 in 464ms, sequenceid=644, compaction requested=true 2023-04-13 14:55:53,943 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:53,943 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:53,947 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@797d2a67[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:53,949 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:53,949 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,950 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,950 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:53,950 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:53,950 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:53,951 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:53,951 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:53,951 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:53,952 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,954 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:53,954 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:53,954 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:53,954 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:53,954 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:53,954 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:53,954 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:53,955 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,955 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,956 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:53,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,956 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,963 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:53,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:53,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:53,963 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:53,963 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:53,964 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,964 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,964 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:53,964 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:53,964 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,964 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:53,964 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:53,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:53,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:53,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:53,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:53,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:53,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:54,063 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:54,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:54,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:54,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:54,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:54,067 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:54,067 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:54,067 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:54,067 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:54,067 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:54,068 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:54,069 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,069 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:54,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:54,069 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,069 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:54,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:54,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:54,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:54,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:54,070 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:54,071 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:54,071 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:54,071 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:54,071 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:54,071 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:54,071 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:54,071 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 14:55:54,071 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:54,071 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:54,072 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:54,072 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:54,072 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:54,072 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:54,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:54,076 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:54,080 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/8045de5a1d8f4669986b63e03e74fde6 2023-04-13 14:55:54,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:54,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397814103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:54,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397814107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:54,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397814108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:54,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397814209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:54,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397814214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:54,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397814214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:54,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397814416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:54,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397814418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:54,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397814419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,494 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/0ac2681c084d459bb73b0e9a6877aabd 2023-04-13 14:55:54,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:54,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397814722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:54,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397814724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:54,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397814725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,904 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/0aab40f0fcfe47e7809938bd3f571be8 2023-04-13 14:55:54,908 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/8045de5a1d8f4669986b63e03e74fde6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8045de5a1d8f4669986b63e03e74fde6 2023-04-13 14:55:54,911 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8045de5a1d8f4669986b63e03e74fde6, entries=150, sequenceid=659, filesize=11.9 K 2023-04-13 14:55:54,912 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/0ac2681c084d459bb73b0e9a6877aabd as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/0ac2681c084d459bb73b0e9a6877aabd 2023-04-13 14:55:54,915 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/0ac2681c084d459bb73b0e9a6877aabd, entries=150, sequenceid=659, filesize=11.9 K 2023-04-13 14:55:54,915 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/0aab40f0fcfe47e7809938bd3f571be8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0aab40f0fcfe47e7809938bd3f571be8 2023-04-13 14:55:54,918 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0aab40f0fcfe47e7809938bd3f571be8, entries=150, sequenceid=659, filesize=11.9 K 2023-04-13 14:55:54,919 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 46dced31cbc65aff91a95c238f6401f0 in 848ms, sequenceid=659, compaction requested=true 2023-04-13 14:55:54,919 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:54,919 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:54,919 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:54,919 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:54,919 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,919 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:54,919 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:54,920 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:54,920 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:54,921 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:54,921 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:54,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55dc2ee2[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:54,924 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:54,924 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:54,925 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:54,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:54,925 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:54,925 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:54,925 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:54,928 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,928 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:54,930 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:54,931 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:54,931 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:54,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:54,933 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:54,933 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:54,933 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:54,933 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:54,933 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:54,934 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:54,935 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:54,935 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:54,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:54,935 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:54,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,936 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:54,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,944 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,944 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:54,944 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:54,944 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:54,944 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:54,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:54,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:54,944 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:54,944 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:54,944 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:54,944 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:54,944 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:54,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:54,944 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:54,944 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:54,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:54,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:55,045 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:55,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:55,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:55,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:55,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:55,048 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:55,048 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:55,048 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:55,048 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:55,048 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,048 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,050 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:55,050 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:55,051 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,051 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:55,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:55,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:55,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:55,057 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:55,057 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:55,057 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:55,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:55,058 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:55,058 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:55,058 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:55,058 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 14:55:55,058 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:55,059 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:55,059 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:55,059 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:55,059 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:55,059 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:55,072 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=676 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/8c157bc6f0f544f3b25bc31edcbf1f6a 2023-04-13 14:55:55,087 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=676 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/1fbe698286a74c7bbaf8bd69d7d00d05 2023-04-13 14:55:55,106 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=676 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/e090a6ca889a4b76be6708d127dd8fbc 2023-04-13 14:55:55,113 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/8c157bc6f0f544f3b25bc31edcbf1f6a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8c157bc6f0f544f3b25bc31edcbf1f6a 2023-04-13 14:55:55,118 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8c157bc6f0f544f3b25bc31edcbf1f6a, entries=150, sequenceid=676, filesize=11.9 K 2023-04-13 14:55:55,119 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/1fbe698286a74c7bbaf8bd69d7d00d05 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/1fbe698286a74c7bbaf8bd69d7d00d05 2023-04-13 14:55:55,122 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/1fbe698286a74c7bbaf8bd69d7d00d05, entries=150, sequenceid=676, filesize=11.9 K 2023-04-13 14:55:55,123 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/e090a6ca889a4b76be6708d127dd8fbc as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/e090a6ca889a4b76be6708d127dd8fbc 2023-04-13 14:55:55,128 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/e090a6ca889a4b76be6708d127dd8fbc, entries=150, sequenceid=676, filesize=11.9 K 2023-04-13 14:55:55,128 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 46dced31cbc65aff91a95c238f6401f0 in 70ms, sequenceid=676, compaction requested=true 2023-04-13 14:55:55,128 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:55,128 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:55,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:55,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:55,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:55,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:55,130 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:55,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:55,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1761b707[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:55,132 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:55,132 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:55,133 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,133 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:55,136 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:55,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:55,141 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:55,141 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:55,141 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:55,141 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:55,141 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:55,143 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,143 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:55,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,143 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:55,145 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,148 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,152 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:55,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:55,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:55,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:55,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:55,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:55,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,152 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:55,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:55,152 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:55,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:55,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:55,241 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:55:55,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:55,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:55,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:55,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:55,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:55,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:55,249 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/cb410ad29b014af7acf71f75b3f5768f 2023-04-13 14:55:55,252 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:55,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:55,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:55,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:55,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:55,260 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:55,260 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:55,260 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:55,260 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:55,261 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,261 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,262 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:55,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:55,262 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,263 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,263 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:55,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:55,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:55,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:55,272 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/7d64f8b4b3894175934ed42fdcab128f 2023-04-13 14:55:55,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:55,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:55,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:55,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:55,272 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:55,272 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:55,272 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:55,272 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:55,273 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:55,273 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:55,273 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:55,273 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,273 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:55,273 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:55,274 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,274 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:55,275 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,275 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:55,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:55,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4714accb[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:55,277 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:55,277 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,279 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:55,280 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,280 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:55,281 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,281 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:55,281 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:55,281 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:55,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:55,284 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:55,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:55,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:55,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:55,284 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:55,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:55,286 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:55,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:55,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:55,286 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,288 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,290 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:55,290 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:55,290 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,290 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,290 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,290 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,290 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,290 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,290 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:55,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:55,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:55,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:55,291 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:55,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:55,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:55,292 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:55,292 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:55,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397755289, max timestamp=1681397755305Num uniques 859; 2023-04-13 14:55:55,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397755289, max timestamp=1681397755305Num uniques 859; 2023-04-13 14:55:55,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:55,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:55,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397755289, max timestamp=1681397755305Num uniques 859; 2023-04-13 14:55:55,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:55,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:55,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397815341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:55,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397815351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:55,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397815352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,392 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:55,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:55,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:55,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:55,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:55,395 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:55,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:55,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:55,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:55,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,396 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,401 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,401 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:55,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:55,401 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,401 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:55,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:55,402 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:55,402 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:55,402 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:55,402 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:55,402 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:55,403 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:55,403 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:55,403 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:55,403 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:55,403 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:55,403 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:55,403 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,403 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:55,403 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:55,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:55,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:55,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57aa985c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:55,406 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:55,406 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,407 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,407 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:55,407 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:55,407 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:55,407 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,407 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:55,408 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:55,408 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:55,408 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,408 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:55,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:55,410 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:55,410 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:55,410 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:55,410 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:55,410 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:55,413 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,413 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,413 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:55,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:55,413 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:55,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:55,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:55,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:55,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:55,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:55,417 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:55,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:55,417 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:55,417 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:55,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:55,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:55,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397815453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:55,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397815460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:55,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397815460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,517 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:55,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:55,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:55,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:55,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:55,521 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:55,521 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:55,521 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:55,521 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:55,521 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,522 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,523 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:55,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:55,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:55,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:55,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:55,524 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:55,524 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:55,524 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:55,525 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:55,525 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:55,525 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:55,525 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:55,530 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:55,531 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,531 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:55,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:55,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63b51775[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:55,532 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:55,532 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,533 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,533 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:55,533 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:55,533 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:55,533 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,533 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:55,535 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:55,535 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:55,535 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:55,535 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:55,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:55,537 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:55,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:55,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:55,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:55,538 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:55,540 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:55,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:55,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:55,541 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,542 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,545 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,545 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,545 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,545 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:55,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:55,546 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:55,546 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:55,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:55,545 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:55,545 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,545 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:55,546 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:55,546 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:55,546 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:55,646 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:55,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:55,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:55,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:55,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:55,649 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:55,649 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:55,649 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:55,649 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:55,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,651 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:55,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:55,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:55,651 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,652 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,652 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:55,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:55,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:55,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:55,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:55,653 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:55,653 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:55,654 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:55,654 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:55,655 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,655 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:55,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:55,657 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:55,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79f64d8b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:55,657 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,658 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,658 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,658 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:55,658 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:55,659 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:55,659 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:55,663 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:55,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:55,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:55,666 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:55,667 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:55,667 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:55,667 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:55,667 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:55,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:55,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397815662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,668 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,668 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:55,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:55,669 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,669 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,670 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:55,672 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:55,672 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:55,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:55,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:55,672 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,672 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,672 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,672 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,672 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,672 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:55,672 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:55,672 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:55,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:55,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:55,672 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:55,672 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:55,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:55,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397815665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:55,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397815666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,695 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/3f6cf16772984cebaa0352bc296d6bd8 2023-04-13 14:55:55,699 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/cb410ad29b014af7acf71f75b3f5768f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/cb410ad29b014af7acf71f75b3f5768f 2023-04-13 14:55:55,703 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/cb410ad29b014af7acf71f75b3f5768f, entries=200, sequenceid=687, filesize=14.3 K 2023-04-13 14:55:55,704 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/7d64f8b4b3894175934ed42fdcab128f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7d64f8b4b3894175934ed42fdcab128f 2023-04-13 14:55:55,706 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7d64f8b4b3894175934ed42fdcab128f, entries=100, sequenceid=687, filesize=9.5 K 2023-04-13 14:55:55,707 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/3f6cf16772984cebaa0352bc296d6bd8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/3f6cf16772984cebaa0352bc296d6bd8 2023-04-13 14:55:55,710 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/3f6cf16772984cebaa0352bc296d6bd8, entries=100, sequenceid=687, filesize=9.5 K 2023-04-13 14:55:55,711 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 46dced31cbc65aff91a95c238f6401f0 in 470ms, sequenceid=687, compaction requested=true 2023-04-13 14:55:55,711 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:55,711 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:55,711 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 14:55:55,711 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 14:55:55,711 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:55,711 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:55,713 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64252 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 14:55:55,713 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/A is initiating minor compaction (all files) 2023-04-13 14:55:55,713 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59372 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 14:55:55,713 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/A in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:55,713 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/B is initiating minor compaction (all files) 2023-04-13 14:55:55,713 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/491351f3b8dd421c83b5b56f2cfdc754, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5e6d0757994243c39fcc5b88611e95c6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8045de5a1d8f4669986b63e03e74fde6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8c157bc6f0f544f3b25bc31edcbf1f6a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/cb410ad29b014af7acf71f75b3f5768f] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=62.7 K 2023-04-13 14:55:55,713 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/B in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:55,713 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/26f60155853a413bbbd1c3619b35408c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6bff03f298474964ae76f75f96c71d47, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/0ac2681c084d459bb73b0e9a6877aabd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/1fbe698286a74c7bbaf8bd69d7d00d05, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7d64f8b4b3894175934ed42fdcab128f] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=58.0 K 2023-04-13 14:55:55,713 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 491351f3b8dd421c83b5b56f2cfdc754, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1681397752510 2023-04-13 14:55:55,713 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 26f60155853a413bbbd1c3619b35408c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1681397752510 2023-04-13 14:55:55,714 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 5e6d0757994243c39fcc5b88611e95c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1681397752579 2023-04-13 14:55:55,714 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 6bff03f298474964ae76f75f96c71d47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1681397752579 2023-04-13 14:55:55,714 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 8045de5a1d8f4669986b63e03e74fde6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1681397753736 2023-04-13 14:55:55,714 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 0ac2681c084d459bb73b0e9a6877aabd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1681397753736 2023-04-13 14:55:55,714 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 8c157bc6f0f544f3b25bc31edcbf1f6a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=676, earliestPutTs=1681397754102 2023-04-13 14:55:55,714 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 1fbe698286a74c7bbaf8bd69d7d00d05, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=676, earliestPutTs=1681397754102 2023-04-13 14:55:55,715 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting cb410ad29b014af7acf71f75b3f5768f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681397755235 2023-04-13 14:55:55,715 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 7d64f8b4b3894175934ed42fdcab128f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681397755240 2023-04-13 14:55:55,723 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#B#compaction#467 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:55,723 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#A#compaction#466 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:55,740 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/900a81d8a9744abcb16aa3fcb4aa65d2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/900a81d8a9744abcb16aa3fcb4aa65d2 2023-04-13 14:55:55,740 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/2a24049fed614357b971eea720a9eba6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/2a24049fed614357b971eea720a9eba6 2023-04-13 14:55:55,743 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/B of 46dced31cbc65aff91a95c238f6401f0 into 2a24049fed614357b971eea720a9eba6(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:55,743 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/A of 46dced31cbc65aff91a95c238f6401f0 into 900a81d8a9744abcb16aa3fcb4aa65d2(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:55,744 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:55,744 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:55,744 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/B, priority=11, startTime=1681397755711; duration=0sec 2023-04-13 14:55:55,744 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/A, priority=11, startTime=1681397755711; duration=0sec 2023-04-13 14:55:55,744 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:55,744 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:55,744 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 14:55:55,745 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71490 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 14:55:55,745 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/C is initiating minor compaction (all files) 2023-04-13 14:55:55,745 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/C in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:55,745 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/f3a1536d2aac415faa8f5ef851a2cfaf, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/566509147a044f17ae2e73610c32811e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/d7426f1057dd47ec8c442e5604dcaf68, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0aab40f0fcfe47e7809938bd3f571be8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/e090a6ca889a4b76be6708d127dd8fbc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/3f6cf16772984cebaa0352bc296d6bd8] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=69.8 K 2023-04-13 14:55:55,746 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting f3a1536d2aac415faa8f5ef851a2cfaf, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681397752019 2023-04-13 14:55:55,746 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 566509147a044f17ae2e73610c32811e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1681397752510 2023-04-13 14:55:55,746 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting d7426f1057dd47ec8c442e5604dcaf68, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1681397752579 2023-04-13 14:55:55,746 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 0aab40f0fcfe47e7809938bd3f571be8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1681397753736 2023-04-13 14:55:55,747 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting e090a6ca889a4b76be6708d127dd8fbc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=676, earliestPutTs=1681397754102 2023-04-13 14:55:55,747 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 3f6cf16772984cebaa0352bc296d6bd8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681397755240 2023-04-13 14:55:55,757 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#C#compaction#468 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:55,770 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/8277414a0932446fb476434c321cd8a3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/8277414a0932446fb476434c321cd8a3 2023-04-13 14:55:55,773 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:55,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:55,775 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/C of 46dced31cbc65aff91a95c238f6401f0 into 8277414a0932446fb476434c321cd8a3(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:55,775 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:55,775 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/C, priority=10, startTime=1681397755711; duration=0sec 2023-04-13 14:55:55,775 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:55,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:55,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:55,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:55,791 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:55,791 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:55,791 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:55,791 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:55,791 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,791 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,792 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,792 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:55,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:55,792 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,792 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:55,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:55,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:55,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:55,793 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:55,793 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:55,793 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:55,796 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:55,796 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:55,796 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:55,796 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:55,796 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-13 14:55:55,797 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:55,797 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:55,797 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:55,797 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:55,797 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:55,797 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:55,808 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=724 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/230f0c9fdda140d9b5724c2ced264c3f 2023-04-13 14:55:55,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:55,975 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:55,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:55,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397815985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:55,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397815991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:55,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:55,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397815992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:56,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397816093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:56,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397816095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:56,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397816099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,222 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=724 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/9eb3d99eddc14750a1c24467e4d7dd32 2023-04-13 14:55:56,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:56,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397816303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:56,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:56,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397816303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397816304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:56,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:56,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397816611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397816611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:56,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397816612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,631 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=724 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/03939d8e73714b2686f1727c6e829eda 2023-04-13 14:55:56,635 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/230f0c9fdda140d9b5724c2ced264c3f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/230f0c9fdda140d9b5724c2ced264c3f 2023-04-13 14:55:56,638 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/230f0c9fdda140d9b5724c2ced264c3f, entries=150, sequenceid=724, filesize=11.9 K 2023-04-13 14:55:56,639 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/9eb3d99eddc14750a1c24467e4d7dd32 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/9eb3d99eddc14750a1c24467e4d7dd32 2023-04-13 14:55:56,642 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/9eb3d99eddc14750a1c24467e4d7dd32, entries=150, sequenceid=724, filesize=11.9 K 2023-04-13 14:55:56,643 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/03939d8e73714b2686f1727c6e829eda as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/03939d8e73714b2686f1727c6e829eda 2023-04-13 14:55:56,646 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/03939d8e73714b2686f1727c6e829eda, entries=150, sequenceid=724, filesize=11.9 K 2023-04-13 14:55:56,646 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 46dced31cbc65aff91a95c238f6401f0 in 850ms, sequenceid=724, compaction requested=false 2023-04-13 14:55:56,646 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:56,646 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:56,646 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:56,647 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:56,647 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,647 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:56,647 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:56,649 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:56,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:56,649 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:56,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:56,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:56,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:56,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b66d48a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:56,651 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:56,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:56,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:56,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:56,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:56,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:56,652 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,652 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:56,653 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:56,653 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,653 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:56,653 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:56,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:56,655 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:56,655 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:56,655 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:56,655 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:56,655 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:56,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:56,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:56,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:56,657 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:56,657 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:56,658 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:56,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:56,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:56,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:56,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:56,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:56,662 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:56,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:56,662 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:56,661 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:56,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:56,662 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:56,662 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:56,662 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:56,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:56,762 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:56,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:56,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:56,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:56,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:56,765 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:56,765 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:56,765 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:56,765 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:56,765 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:56,765 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:56,767 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:56,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:56,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:56,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:56,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:56,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:56,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:56,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:56,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:56,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:56,768 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:56,768 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:56,768 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 14:55:56,769 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:56,769 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:56,769 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:56,769 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:56,769 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:56,769 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:56,782 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/14e144c2107c49b783de4d235940cf45 2023-04-13 14:55:56,793 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/6617fdf157ce440e9af20eb27db6b0e3 2023-04-13 14:55:57,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:57,123 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:57,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681397757145, max timestamp=1681397757164Num uniques 859; 2023-04-13 14:55:57,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:57,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681397757145, max timestamp=1681397757164Num uniques 859; 2023-04-13 14:55:57,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:57,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681397757145, max timestamp=1681397757164Num uniques 859; 2023-04-13 14:55:57,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:57,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397817191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397817193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397817194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,204 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/fb237007775e4e96ae3ae6e114ca45f0 2023-04-13 14:55:57,207 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/14e144c2107c49b783de4d235940cf45 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/14e144c2107c49b783de4d235940cf45 2023-04-13 14:55:57,210 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/14e144c2107c49b783de4d235940cf45, entries=150, sequenceid=735, filesize=11.9 K 2023-04-13 14:55:57,211 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/6617fdf157ce440e9af20eb27db6b0e3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6617fdf157ce440e9af20eb27db6b0e3 2023-04-13 14:55:57,214 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6617fdf157ce440e9af20eb27db6b0e3, entries=150, sequenceid=735, filesize=11.9 K 2023-04-13 14:55:57,215 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/fb237007775e4e96ae3ae6e114ca45f0 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/fb237007775e4e96ae3ae6e114ca45f0 2023-04-13 14:55:57,218 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/fb237007775e4e96ae3ae6e114ca45f0, entries=150, sequenceid=735, filesize=11.9 K 2023-04-13 14:55:57,219 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 46dced31cbc65aff91a95c238f6401f0 in 451ms, sequenceid=735, compaction requested=true 2023-04-13 14:55:57,219 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:57,219 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:57,219 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:57,219 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:57,219 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,219 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:57,219 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:57,221 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:57,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,221 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:57,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@188dc7c2[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:57,223 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:57,223 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,227 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,227 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:57,227 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:57,227 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:57,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:57,229 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:57,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:57,229 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:57,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:57,231 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:57,231 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:57,231 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:57,231 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:57,231 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:57,233 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,233 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:57,233 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:57,233 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:57,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:57,235 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,237 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:57,237 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:57,238 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:57,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:57,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:57,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:57,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:57,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:57,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,238 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,237 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:57,238 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:57,238 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:57,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:57,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:57,302 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-13 14:55:57,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:57,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:57,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:57,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:57,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:57,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:57,312 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=770 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/d89a1814c3b342e4ae5ed55d3d229a08 2023-04-13 14:55:57,321 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=770 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/29906e6dd5ae4ae58312339db959be82 2023-04-13 14:55:57,329 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=770 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/71cb2a6865684d8d96880f1db05e16af 2023-04-13 14:55:57,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/d89a1814c3b342e4ae5ed55d3d229a08 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/d89a1814c3b342e4ae5ed55d3d229a08 2023-04-13 14:55:57,335 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/d89a1814c3b342e4ae5ed55d3d229a08, entries=200, sequenceid=770, filesize=14.3 K 2023-04-13 14:55:57,336 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/29906e6dd5ae4ae58312339db959be82 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/29906e6dd5ae4ae58312339db959be82 2023-04-13 14:55:57,338 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:57,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397817329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:57,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397817330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/29906e6dd5ae4ae58312339db959be82, entries=150, sequenceid=770, filesize=11.9 K 2023-04-13 14:55:57,341 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/71cb2a6865684d8d96880f1db05e16af as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/71cb2a6865684d8d96880f1db05e16af 2023-04-13 14:55:57,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:57,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:57,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:57,342 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:57,342 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:57,342 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:57,342 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:57,342 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,342 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:57,343 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/71cb2a6865684d8d96880f1db05e16af, entries=150, sequenceid=770, filesize=11.9 K 2023-04-13 14:55:57,344 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 46dced31cbc65aff91a95c238f6401f0 in 42ms, sequenceid=770, compaction requested=true 2023-04-13 14:55:57,344 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:57,344 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:57,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:57,344 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:57,344 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:55:57,344 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:55:57,344 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:57,344 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:57,345 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:57,345 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,345 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:57,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:57,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:57,345 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52240 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:55:57,345 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:55:57,346 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/A is initiating minor compaction (all files) 2023-04-13 14:55:57,346 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/B is initiating minor compaction (all files) 2023-04-13 14:55:57,346 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/A in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:57,346 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/B in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:57,346 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/900a81d8a9744abcb16aa3fcb4aa65d2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/230f0c9fdda140d9b5724c2ced264c3f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/14e144c2107c49b783de4d235940cf45, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/d89a1814c3b342e4ae5ed55d3d229a08] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=51.0 K 2023-04-13 14:55:57,346 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/2a24049fed614357b971eea720a9eba6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/9eb3d99eddc14750a1c24467e4d7dd32, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6617fdf157ce440e9af20eb27db6b0e3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/29906e6dd5ae4ae58312339db959be82] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=48.6 K 2023-04-13 14:55:57,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:57,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:57,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:57,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:57,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:57,346 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 900a81d8a9744abcb16aa3fcb4aa65d2, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681397754102 2023-04-13 14:55:57,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:57,346 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:57,346 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 2a24049fed614357b971eea720a9eba6, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681397754102 2023-04-13 14:55:57,346 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:57,347 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:57,347 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:55:57,347 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 230f0c9fdda140d9b5724c2ced264c3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=724, earliestPutTs=1681397755337 2023-04-13 14:55:57,347 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 9eb3d99eddc14750a1c24467e4d7dd32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=724, earliestPutTs=1681397755337 2023-04-13 14:55:57,347 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 14e144c2107c49b783de4d235940cf45, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1681397755981 2023-04-13 14:55:57,347 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:57,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:57,347 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting d89a1814c3b342e4ae5ed55d3d229a08, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1681397757190 2023-04-13 14:55:57,347 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:57,347 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 6617fdf157ce440e9af20eb27db6b0e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1681397755981 2023-04-13 14:55:57,347 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:57,347 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:57,348 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:57,348 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:57,348 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 29906e6dd5ae4ae58312339db959be82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1681397757191 2023-04-13 14:55:57,348 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:57,357 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#A#compaction#479 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:57,359 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#B#compaction#480 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:57,372 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/7731556ca91a4fb096547830a90e1342 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/7731556ca91a4fb096547830a90e1342 2023-04-13 14:55:57,373 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/ceb6c75ea7c1416da590366316427367 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/ceb6c75ea7c1416da590366316427367 2023-04-13 14:55:57,375 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=781 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/297999a17f2f4ff8aeea4195bb902c8c 2023-04-13 14:55:57,377 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/A of 46dced31cbc65aff91a95c238f6401f0 into 7731556ca91a4fb096547830a90e1342(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:57,377 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:57,377 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/A, priority=12, startTime=1681397757344; duration=0sec 2023-04-13 14:55:57,378 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:57,378 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:55:57,378 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/B of 46dced31cbc65aff91a95c238f6401f0 into ceb6c75ea7c1416da590366316427367(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:57,378 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:57,378 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/B, priority=12, startTime=1681397757344; duration=0sec 2023-04-13 14:55:57,378 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:57,379 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49766 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:55:57,379 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/C is initiating minor compaction (all files) 2023-04-13 14:55:57,379 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/C in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:57,379 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/8277414a0932446fb476434c321cd8a3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/03939d8e73714b2686f1727c6e829eda, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/fb237007775e4e96ae3ae6e114ca45f0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/71cb2a6865684d8d96880f1db05e16af] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=48.6 K 2023-04-13 14:55:57,380 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 8277414a0932446fb476434c321cd8a3, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681397754102 2023-04-13 14:55:57,380 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 03939d8e73714b2686f1727c6e829eda, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=724, earliestPutTs=1681397755337 2023-04-13 14:55:57,380 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting fb237007775e4e96ae3ae6e114ca45f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1681397755981 2023-04-13 14:55:57,380 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 71cb2a6865684d8d96880f1db05e16af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1681397757191 2023-04-13 14:55:57,392 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=781 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/6ad64fd6298f4784aa4b0dc6e8b2b282 2023-04-13 14:55:57,394 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#C#compaction#482 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:57,423 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/b38ee05663ae4e9ba05f3366d5c5ae28 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/b38ee05663ae4e9ba05f3366d5c5ae28 2023-04-13 14:55:57,427 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/C of 46dced31cbc65aff91a95c238f6401f0 into b38ee05663ae4e9ba05f3366d5c5ae28(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:57,427 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:57,427 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/C, priority=12, startTime=1681397757345; duration=0sec 2023-04-13 14:55:57,427 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:57,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397757435, max timestamp=1681397757466Num uniques 859; 2023-04-13 14:55:57,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397757435, max timestamp=1681397757466Num uniques 859; 2023-04-13 14:55:57,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:57,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397817466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397757457, max timestamp=1681397757466Num uniques 859; 2023-04-13 14:55:57,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:55:57,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:57,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397817509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397817519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397817570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397817620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397817627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397817775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,815 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=781 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/8ebafa09f47d47609286477b281803f0 2023-04-13 14:55:57,819 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/297999a17f2f4ff8aeea4195bb902c8c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/297999a17f2f4ff8aeea4195bb902c8c 2023-04-13 14:55:57,823 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/297999a17f2f4ff8aeea4195bb902c8c, entries=200, sequenceid=781, filesize=14.3 K 2023-04-13 14:55:57,824 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/6ad64fd6298f4784aa4b0dc6e8b2b282 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6ad64fd6298f4784aa4b0dc6e8b2b282 2023-04-13 14:55:57,827 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6ad64fd6298f4784aa4b0dc6e8b2b282, entries=150, sequenceid=781, filesize=11.9 K 2023-04-13 14:55:57,828 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/8ebafa09f47d47609286477b281803f0 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/8ebafa09f47d47609286477b281803f0 2023-04-13 14:55:57,831 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/8ebafa09f47d47609286477b281803f0, entries=150, sequenceid=781, filesize=11.9 K 2023-04-13 14:55:57,831 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 46dced31cbc65aff91a95c238f6401f0 in 484ms, sequenceid=781, compaction requested=false 2023-04-13 14:55:57,831 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:57,831 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,833 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,834 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,834 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:57,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:57,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53486519[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:57,836 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:57,836 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,837 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:57,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:57,837 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,837 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:57,837 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=136.42 KB heapSize=385.38 KB 2023-04-13 14:55:57,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:57,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:57,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:57,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:57,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:57,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:57,840 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:57,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:57,845 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:57,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:57,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:57,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:57,845 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:57,848 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,848 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:57,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,849 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,850 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,852 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:57,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:57,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:57,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:57,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:57,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,853 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:57,853 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:57,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:57,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:57,854 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:57,854 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,854 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:57,854 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:57,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:57,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:57,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397817863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397817870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,954 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:57,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:57,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:57,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:57,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:57,957 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:57,958 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:57,958 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:57,958 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:57,958 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,958 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:57,959 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:57,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:57,959 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,960 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:57,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:57,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:57,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:57,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:57,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:57,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:57,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:57,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:57,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:57,961 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:57,961 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:57,961 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:57,961 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:57,964 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:57,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:57,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1899e0ee[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:57,966 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:57,966 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,968 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,968 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:57,968 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,968 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:57,968 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:57,968 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:57,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:57,969 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:57,969 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:57,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:57,972 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:57,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:57,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:57,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:57,972 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:57,973 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,973 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:57,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,973 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:57,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:57,974 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:57,975 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,977 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:57,977 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:57,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:57,977 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:57,977 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:57,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:57,977 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:57,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:57,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:57,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:57,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:57,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:57,978 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:57,978 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:57,978 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:57,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397817974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:57,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:57,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397817974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,078 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:58,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:58,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:58,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:58,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:58,082 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:58,082 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:58,082 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:58,082 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:58,082 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,082 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,085 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,085 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,085 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:58,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:58,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397818080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,085 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:58,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:58,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:58,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:58,087 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:58,087 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:58,087 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:58,087 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:58,087 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:58,087 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:58,087 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:58,087 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:58,087 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:58,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:58,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:58,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:58,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:58,089 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,089 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:58,089 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,089 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:58,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:58,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3576a5f3[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:58,091 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:58,091 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,092 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,092 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:58,092 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:58,092 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:58,092 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,092 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:58,097 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,097 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:58,098 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:58,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:58,098 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:58,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:58,100 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:58,100 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:58,100 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:58,100 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:58,100 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:58,102 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,102 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:58,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:58,102 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:58,102 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,103 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,105 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:58,106 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:58,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:58,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:58,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:58,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:58,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:58,106 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,106 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,106 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,106 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,106 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,106 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,106 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,106 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:58,106 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:58,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:58,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:58,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:58,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397818185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:58,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397818185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,207 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:58,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:58,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:58,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:58,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:58,210 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:58,210 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:58,210 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:58,210 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:58,211 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,211 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,212 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:58,212 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:58,212 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,212 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:58,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:58,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:58,214 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:58,214 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:58,214 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:58,214 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:58,214 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:58,214 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:58,214 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:58,214 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:58,214 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:58,214 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:58,214 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:58,214 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,214 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:58,214 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:58,217 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,217 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:58,217 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,217 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:58,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:58,219 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:58,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4333c748[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:58,219 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,220 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:58,220 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,220 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:58,222 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,222 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:58,222 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:58,222 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:58,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:58,225 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:58,225 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:58,225 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:58,225 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:58,225 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:58,226 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,226 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,226 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:58,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,227 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,228 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,231 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,231 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,231 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,231 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,231 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,231 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,231 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:58,231 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:58,232 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:58,231 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:58,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:58,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:58,232 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:58,232 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:58,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:58,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:58,249 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=820 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/b702741bffc0486a9ad65e4004d76633 2023-04-13 14:55:58,260 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=820 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/fef5b963d465407ca7d9804153e46701 2023-04-13 14:55:58,270 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=820 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/a468ae2d57514cd7bde4852ef1b295fd 2023-04-13 14:55:58,274 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/b702741bffc0486a9ad65e4004d76633 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/b702741bffc0486a9ad65e4004d76633 2023-04-13 14:55:58,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/b702741bffc0486a9ad65e4004d76633, entries=200, sequenceid=820, filesize=14.3 K 2023-04-13 14:55:58,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/fef5b963d465407ca7d9804153e46701 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/fef5b963d465407ca7d9804153e46701 2023-04-13 14:55:58,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/fef5b963d465407ca7d9804153e46701, entries=150, sequenceid=820, filesize=11.9 K 2023-04-13 14:55:58,282 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/a468ae2d57514cd7bde4852ef1b295fd as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/a468ae2d57514cd7bde4852ef1b295fd 2023-04-13 14:55:58,287 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/a468ae2d57514cd7bde4852ef1b295fd, entries=150, sequenceid=820, filesize=11.9 K 2023-04-13 14:55:58,287 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~385.33 KB/394576, currentSize=46.96 KB/48090 for 46dced31cbc65aff91a95c238f6401f0 in 450ms, sequenceid=820, compaction requested=true 2023-04-13 14:55:58,288 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:58,288 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:58,288 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:58,288 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:58,288 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:58,288 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:58,290 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42630 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:58,290 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:58,290 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/A is initiating minor compaction (all files) 2023-04-13 14:55:58,290 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/B is initiating minor compaction (all files) 2023-04-13 14:55:58,290 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/A in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:58,290 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/B in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:58,290 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/7731556ca91a4fb096547830a90e1342, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/297999a17f2f4ff8aeea4195bb902c8c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/b702741bffc0486a9ad65e4004d76633] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=41.6 K 2023-04-13 14:55:58,290 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/ceb6c75ea7c1416da590366316427367, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6ad64fd6298f4784aa4b0dc6e8b2b282, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/fef5b963d465407ca7d9804153e46701] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=36.9 K 2023-04-13 14:55:58,291 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 7731556ca91a4fb096547830a90e1342, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1681397757191 2023-04-13 14:55:58,291 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting ceb6c75ea7c1416da590366316427367, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1681397757191 2023-04-13 14:55:58,291 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 297999a17f2f4ff8aeea4195bb902c8c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=781, earliestPutTs=1681397757314 2023-04-13 14:55:58,291 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 6ad64fd6298f4784aa4b0dc6e8b2b282, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=781, earliestPutTs=1681397757314 2023-04-13 14:55:58,291 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting b702741bffc0486a9ad65e4004d76633, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1681397757483 2023-04-13 14:55:58,292 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting fef5b963d465407ca7d9804153e46701, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1681397757483 2023-04-13 14:55:58,298 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#B#compaction#487 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:58,300 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#A#compaction#488 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:58,332 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:58,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:58,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:58,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:58,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:58,336 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:58,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:58,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:58,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:58,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,338 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:58,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:58,339 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,339 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:58,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:58,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:58,340 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:58,340 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:58,340 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:58,344 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:58,344 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:58,344 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:58,344 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:58,344 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:55:58,344 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:58,344 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:58,344 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:58,344 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:58,344 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:58,345 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:58,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:58,498 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:58,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397758520, max timestamp=1681397758539Num uniques 859; 2023-04-13 14:55:58,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397758520, max timestamp=1681397758539Num uniques 859; 2023-04-13 14:55:58,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:58,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681397758520, max timestamp=1681397758539Num uniques 859; 2023-04-13 14:55:58,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:58,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 14:55:58,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:58,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397818586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:58,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397818588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:58,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397818595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:58,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397818696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:58,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397818706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,723 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/98d9c512e2f94c8aadfa47ccfc56a298 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/98d9c512e2f94c8aadfa47ccfc56a298 2023-04-13 14:55:58,727 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/0b05c80df0874bc2a02b92a5195a0d00 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/0b05c80df0874bc2a02b92a5195a0d00 2023-04-13 14:55:58,728 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/B of 46dced31cbc65aff91a95c238f6401f0 into 98d9c512e2f94c8aadfa47ccfc56a298(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:58,729 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:58,729 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/B, priority=13, startTime=1681397758288; duration=0sec 2023-04-13 14:55:58,729 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:58,729 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:58,730 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:58,730 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/C is initiating minor compaction (all files) 2023-04-13 14:55:58,730 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/C in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:58,730 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/b38ee05663ae4e9ba05f3366d5c5ae28, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/8ebafa09f47d47609286477b281803f0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/a468ae2d57514cd7bde4852ef1b295fd] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=36.8 K 2023-04-13 14:55:58,730 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting b38ee05663ae4e9ba05f3366d5c5ae28, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1681397757191 2023-04-13 14:55:58,731 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 8ebafa09f47d47609286477b281803f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=781, earliestPutTs=1681397757314 2023-04-13 14:55:58,731 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting a468ae2d57514cd7bde4852ef1b295fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1681397757483 2023-04-13 14:55:58,733 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/A of 46dced31cbc65aff91a95c238f6401f0 into 0b05c80df0874bc2a02b92a5195a0d00(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:58,733 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:58,733 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/A, priority=13, startTime=1681397758288; duration=0sec 2023-04-13 14:55:58,733 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:58,738 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#C#compaction#490 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:58,756 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/c2f1979e11ac40d8b63ebb2203d10e62 2023-04-13 14:55:58,768 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/32b7aa109f294fc5a704e1aba1315edb 2023-04-13 14:55:58,778 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/5ddc4f696ebc4fd9abbd17d43ce7b0a8 2023-04-13 14:55:58,784 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/c2f1979e11ac40d8b63ebb2203d10e62 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/c2f1979e11ac40d8b63ebb2203d10e62 2023-04-13 14:55:58,789 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/c2f1979e11ac40d8b63ebb2203d10e62, entries=150, sequenceid=830, filesize=11.9 K 2023-04-13 14:55:58,790 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/32b7aa109f294fc5a704e1aba1315edb as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/32b7aa109f294fc5a704e1aba1315edb 2023-04-13 14:55:58,793 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/32b7aa109f294fc5a704e1aba1315edb, entries=150, sequenceid=830, filesize=11.9 K 2023-04-13 14:55:58,793 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-13 14:55:58,794 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/5ddc4f696ebc4fd9abbd17d43ce7b0a8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5ddc4f696ebc4fd9abbd17d43ce7b0a8 2023-04-13 14:55:58,798 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5ddc4f696ebc4fd9abbd17d43ce7b0a8, entries=150, sequenceid=830, filesize=11.9 K 2023-04-13 14:55:58,799 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 46dced31cbc65aff91a95c238f6401f0 in 455ms, sequenceid=830, compaction requested=false 2023-04-13 14:55:58,799 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:58,799 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:58,799 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:58,799 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:58,799 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,799 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:58,799 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:58,801 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,801 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:58,801 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,801 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:58,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:58,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5343129a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:58,804 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:58,805 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,806 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,806 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:58,806 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:58,806 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:58,806 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,806 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:58,808 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:58,808 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:58,808 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,808 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:58,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:58,811 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:58,811 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:58,811 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:58,811 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:58,811 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:58,812 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,812 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:58,812 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:58,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:58,813 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,814 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,818 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:58,818 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:58,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:58,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:58,819 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,819 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,820 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,820 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,820 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,820 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,820 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,820 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:58,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:58,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:58,820 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:58,820 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:58,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:55:58,912 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-13 14:55:58,912 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:58,912 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:58,912 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:58,912 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:58,912 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:58,912 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:55:58,920 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:58,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:58,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:58,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:58,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:58,924 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:58,924 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:58,924 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:58,924 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:58,924 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,924 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,928 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:58,928 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,929 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,929 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:58,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:58,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:58,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:58,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:58,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:58,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:58,930 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:58,930 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:58,930 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:58,930 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:58,930 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:58,931 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:58,931 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:58,931 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,931 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:58,931 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:58,933 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:58,933 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:58,933 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,933 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:58,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:58,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40a068c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:58,935 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:58,935 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,936 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,936 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,936 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:58,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,937 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:58,937 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:58,937 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:58,938 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:58,938 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:58,938 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,938 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:58,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:58,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397818930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:58,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:58,941 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:58,941 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:58,941 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:58,941 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:58,941 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:58,943 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,943 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:58,943 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:58,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:58,943 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,945 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:58,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397818940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,954 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:58,954 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:58,954 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,955 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,955 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:58,955 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:58,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:58,955 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,955 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:58,955 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:58,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:58,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:58,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:58,955 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:58,955 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:58,955 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:58,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:59,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:59,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397819041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:59,055 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:59,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397819050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:59,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:59,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:59,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:59,059 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:59,059 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:59,059 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:59,059 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:59,059 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,059 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,060 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:59,060 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:59,061 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,061 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:59,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:59,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:59,063 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:59,063 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:59,063 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:59,064 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:59,064 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:59,064 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,064 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:59,064 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:59,064 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,064 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:59,064 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:59,064 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,064 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:59,064 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:59,066 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d362c3[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:59,068 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:59,068 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,069 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,070 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:59,070 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:59,070 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:59,072 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:59,072 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,072 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:59,072 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:59,072 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:59,072 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:59,072 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:59,073 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:59,073 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:59,073 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:59,073 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:59,074 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,074 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,074 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:59,075 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,076 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,078 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:59,078 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:59,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:59,078 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:59,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:59,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:59,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:59,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:59,079 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:59,079 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:59,079 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,079 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,079 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,079 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,079 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,079 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,079 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,157 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/247d76ef8a1845889f7e51397e782212 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/247d76ef8a1845889f7e51397e782212 2023-04-13 14:55:59,161 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/C of 46dced31cbc65aff91a95c238f6401f0 into 247d76ef8a1845889f7e51397e782212(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:59,161 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:59,161 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/C, priority=13, startTime=1681397758288; duration=0sec 2023-04-13 14:55:59,162 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:59,179 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:59,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:59,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:59,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:59,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:59,182 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:59,182 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:59,182 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:59,182 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:59,183 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,183 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,184 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,184 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:59,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:59,184 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,184 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:59,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:59,186 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:59,186 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:59,186 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,186 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:59,186 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:59,186 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,186 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:59,186 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:59,186 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,186 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:59,186 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:59,188 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,188 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,189 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,189 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:59,190 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:59,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5987fc96[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:59,190 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,191 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,191 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:59,191 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:59,191 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:59,192 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,192 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:59,193 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:59,193 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:59,193 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,193 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:59,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:59,195 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:59,195 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:59,195 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:59,195 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:59,195 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:59,196 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,196 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,197 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:59,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:59,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:59,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,197 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,199 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,203 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,203 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,203 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,203 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,203 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,203 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,203 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,203 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:59,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:59,204 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:59,204 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:59,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:59,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:59,204 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:59,204 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:59,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:59,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:59,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:59,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397819242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:59,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397819256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,304 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:59,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:59,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:59,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:59,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:59,308 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:59,308 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:59,308 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:59,308 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:59,315 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,315 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,316 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:59,316 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,317 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,317 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:59,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:59,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:59,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:59,318 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:59,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:59,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:59,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:59,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:59,319 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,319 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:59,319 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:59,319 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:59,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:59,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:59,319 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:59,320 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,321 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,321 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,321 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,321 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=868 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/ddfd57599d864f0ab8d7f1ecc3db9dc7 2023-04-13 14:55:59,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:59,324 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:59,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@129bf2fe[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:59,324 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,328 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,329 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:59,329 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:59,329 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:59,330 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,330 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:59,332 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:59,332 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,332 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:59,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,332 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:59,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:59,336 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:59,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:59,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:59,337 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:59,337 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:59,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,338 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:59,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,339 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,340 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,345 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=868 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/ff78bad9234548bfa87f061f8b1b3089 2023-04-13 14:55:59,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:59,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:59,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:59,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:59,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:59,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:59,348 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:59,349 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:59,349 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:59,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:59,449 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:59,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:59,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:59,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:59,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:59,453 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:59,453 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:59,453 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:59,453 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:59,453 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,453 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,459 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,459 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:59,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:59,459 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,459 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:59,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:59,461 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:59,461 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:59,461 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,461 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:59,461 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:59,461 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,461 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:59,461 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:59,461 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,461 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:59,461 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:59,462 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,462 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,463 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,463 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:59,464 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:59,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46f1a847[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:59,464 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,465 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,465 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:59,465 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:59,465 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:59,465 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,465 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:59,466 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:59,466 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:59,466 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,466 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:59,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:59,468 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:59,469 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:59,469 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:59,469 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:59,469 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:59,471 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,471 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,471 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:59,472 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,473 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,475 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:59,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:59,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:59,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:59,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:59,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:59,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:59,475 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:59,475 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:59,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:59,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:59,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53300 deadline: 1681397819547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:59,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53268 deadline: 1681397819563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,575 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:59,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:59,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:59,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:59,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:59,578 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:59,578 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:59,578 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:59,578 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:59,579 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,579 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,580 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:59,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:59,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:59,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:59,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:59,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:59,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:59,581 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:59,581 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:59,581 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:59,581 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:59,581 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:59,581 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,581 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:59,582 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:59,582 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,582 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:59,582 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:59,582 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,582 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:59,582 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:59,583 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,583 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,583 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,583 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:59,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:59,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@592b1f20[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:59,585 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:59,585 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,587 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,587 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:59,587 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,587 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:59,587 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:59,587 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:59,588 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:59,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:59,590 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:59,590 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:59,590 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:59,590 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:59,590 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:59,592 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,592 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,592 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,592 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,593 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,595 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,595 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,595 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:59,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:59,595 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:59,596 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:59,596 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:59,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:59,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:59,596 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,596 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,596 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:59,596 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:59,596 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,596 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:59,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:59,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:55:59,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53296 deadline: 1681397819600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,696 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:59,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:59,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:59,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:59,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:59,699 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:59,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:59,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:59,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:59,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,700 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,702 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:59,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:59,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:59,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,703 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:59,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:59,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:59,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:59,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:59,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:59,704 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,704 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:59,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:59,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:59,704 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,704 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:59,704 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:55:59,704 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:59,704 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:59,705 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,705 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:59,705 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:59,706 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:59,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fae3bdf[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:59,708 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:59,708 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,709 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,709 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:59,709 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:59,709 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:59,709 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,709 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:59,710 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,710 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:59,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,710 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:59,711 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:59,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:59,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:59,713 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:59,713 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:59,713 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:59,713 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:59,713 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:59,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,716 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:59,716 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,720 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,722 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:59,722 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,722 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:59,722 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:59,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:59,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:59,722 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,723 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,723 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,723 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,723 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,723 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:59,723 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:59,722 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:59,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:59,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:59,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:59,770 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=868 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/7286a33650874915be4432369df69018 2023-04-13 14:55:59,774 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/ddfd57599d864f0ab8d7f1ecc3db9dc7 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/ddfd57599d864f0ab8d7f1ecc3db9dc7 2023-04-13 14:55:59,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/ddfd57599d864f0ab8d7f1ecc3db9dc7, entries=200, sequenceid=868, filesize=14.3 K 2023-04-13 14:55:59,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/ff78bad9234548bfa87f061f8b1b3089 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/ff78bad9234548bfa87f061f8b1b3089 2023-04-13 14:55:59,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/ff78bad9234548bfa87f061f8b1b3089, entries=150, sequenceid=868, filesize=11.9 K 2023-04-13 14:55:59,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/7286a33650874915be4432369df69018 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/7286a33650874915be4432369df69018 2023-04-13 14:55:59,784 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/7286a33650874915be4432369df69018, entries=150, sequenceid=868, filesize=11.9 K 2023-04-13 14:55:59,785 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 46dced31cbc65aff91a95c238f6401f0 in 873ms, sequenceid=868, compaction requested=true 2023-04-13 14:55:59,785 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:59,786 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:59,786 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:59,786 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:59,786 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:59,786 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:59,787 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40292 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:59,787 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/A is initiating minor compaction (all files) 2023-04-13 14:55:59,787 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:59,787 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/A in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,787 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/B is initiating minor compaction (all files) 2023-04-13 14:55:59,787 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/0b05c80df0874bc2a02b92a5195a0d00, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/c2f1979e11ac40d8b63ebb2203d10e62, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/ddfd57599d864f0ab8d7f1ecc3db9dc7] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=39.3 K 2023-04-13 14:55:59,787 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/B in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,787 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/98d9c512e2f94c8aadfa47ccfc56a298, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/32b7aa109f294fc5a704e1aba1315edb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/ff78bad9234548bfa87f061f8b1b3089] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=37.0 K 2023-04-13 14:55:59,788 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 0b05c80df0874bc2a02b92a5195a0d00, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1681397757483 2023-04-13 14:55:59,788 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 98d9c512e2f94c8aadfa47ccfc56a298, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1681397757483 2023-04-13 14:55:59,788 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting c2f1979e11ac40d8b63ebb2203d10e62, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1681397757850 2023-04-13 14:55:59,788 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 32b7aa109f294fc5a704e1aba1315edb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1681397757850 2023-04-13 14:55:59,789 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting ddfd57599d864f0ab8d7f1ecc3db9dc7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1681397758583 2023-04-13 14:55:59,789 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting ff78bad9234548bfa87f061f8b1b3089, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1681397758583 2023-04-13 14:55:59,795 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#B#compaction#496 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:59,795 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#A#compaction#497 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:59,816 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/30db4d4d151a47788dfb96427c72e3d7 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/30db4d4d151a47788dfb96427c72e3d7 2023-04-13 14:55:59,820 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/B of 46dced31cbc65aff91a95c238f6401f0 into 30db4d4d151a47788dfb96427c72e3d7(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:59,820 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:59,820 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/B, priority=13, startTime=1681397759786; duration=0sec 2023-04-13 14:55:59,820 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:55:59,820 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:55:59,821 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:55:59,821 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/C is initiating minor compaction (all files) 2023-04-13 14:55:59,821 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/C in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,821 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/247d76ef8a1845889f7e51397e782212, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5ddc4f696ebc4fd9abbd17d43ce7b0a8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/7286a33650874915be4432369df69018] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=36.9 K 2023-04-13 14:55:59,822 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 247d76ef8a1845889f7e51397e782212, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1681397757483 2023-04-13 14:55:59,822 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 5ddc4f696ebc4fd9abbd17d43ce7b0a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1681397757850 2023-04-13 14:55:59,822 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 7286a33650874915be4432369df69018, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1681397758583 2023-04-13 14:55:59,823 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:55:59,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:55:59,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:55:59,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:55:59,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:55:59,826 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:55:59,826 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:59,826 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:55:59,826 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:55:59,826 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,826 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,830 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,830 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:59,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:59,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:59,830 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,830 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:55:59,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:55:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:55:59,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:55:59,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:55:59,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:55:59,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:55:59,831 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:55:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:55:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:55:59,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:55:59,832 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 14:55:59,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:55:59,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:59,832 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:55:59,833 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:59,833 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:55:59,833 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:55:59,857 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#C#compaction#499 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:55:59,869 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/baa282af591f48cfac27ea153ffa7c7a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/baa282af591f48cfac27ea153ffa7c7a 2023-04-13 14:55:59,874 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/C of 46dced31cbc65aff91a95c238f6401f0 into baa282af591f48cfac27ea153ffa7c7a(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:55:59,874 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:59,874 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/C, priority=13, startTime=1681397759786; duration=0sec 2023-04-13 14:55:59,874 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:55:59,880 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/5c483f76131b472497d5b76db12b6423 2023-04-13 14:55:59,913 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/7f28c91b1cbb4ba39f11ce2529e441e9 2023-04-13 14:55:59,926 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/2d70fcc33dfe4fc2b443002131b8a312 2023-04-13 14:55:59,930 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/5c483f76131b472497d5b76db12b6423 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5c483f76131b472497d5b76db12b6423 2023-04-13 14:55:59,933 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5c483f76131b472497d5b76db12b6423, entries=150, sequenceid=879, filesize=11.9 K 2023-04-13 14:55:59,935 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/7f28c91b1cbb4ba39f11ce2529e441e9 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7f28c91b1cbb4ba39f11ce2529e441e9 2023-04-13 14:55:59,939 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7f28c91b1cbb4ba39f11ce2529e441e9, entries=150, sequenceid=879, filesize=11.9 K 2023-04-13 14:55:59,940 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/2d70fcc33dfe4fc2b443002131b8a312 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/2d70fcc33dfe4fc2b443002131b8a312 2023-04-13 14:55:59,946 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/2d70fcc33dfe4fc2b443002131b8a312, entries=150, sequenceid=879, filesize=11.9 K 2023-04-13 14:55:59,946 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 46dced31cbc65aff91a95c238f6401f0 in 114ms, sequenceid=879, compaction requested=false 2023-04-13 14:55:59,946 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:55:59,946 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:55:59,947 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:55:59,947 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:55:59,947 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,947 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:55:59,947 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:55:59,949 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,949 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,950 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,950 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:55:59,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:55:59,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@726f22d6[Count = 0] remaining members to acquire global barrier 2023-04-13 14:55:59,951 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:55:59,951 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,952 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,952 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:55:59,953 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:55:59,953 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:55:59,953 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,953 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:55:59,954 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,954 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:55:59,954 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:59,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,954 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:55:59,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:55:59,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:55:59,958 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:55:59,958 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:55:59,958 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:55:59,958 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:55:59,958 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:55:59,960 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,960 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:55:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:55:59,960 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:59,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:55:59,960 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:55:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:55:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,963 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:55:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:55:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,966 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:55:59,966 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:55:59,966 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:55:59,966 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,966 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,966 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:55:59,966 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:55:59,966 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,966 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:55:59,966 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:55:59,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:55:59,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:55:59,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:55:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:55:59,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:55:59,966 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:55:59,966 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:55:59,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:55:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:00,066 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:00,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:00,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:00,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:00,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:00,069 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:00,069 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:00,070 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:00,070 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:00,070 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:00,070 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:00,071 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:00,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:00,071 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:00,071 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,072 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:00,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:00,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:00,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:00,073 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:00,073 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:00,073 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:00,073 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:00,073 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:00,073 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:56:00,073 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. started... 2023-04-13 14:56:00,074 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 14:56:00,074 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:56:00,074 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:00,074 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:56:00,074 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:00,074 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:56:00,074 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:00,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:56:00,076 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. as already flushing 2023-04-13 14:56:00,085 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=889 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/3f7a7ac61e8e4514a72c89069c774194 2023-04-13 14:56:00,095 DEBUG [Thread-2929] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x067ea274 to 127.0.0.1:62626 2023-04-13 14:56:00,095 DEBUG [Thread-2929] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:00,099 DEBUG [Thread-2933] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x659c68ab to 127.0.0.1:62626 2023-04-13 14:56:00,099 DEBUG [Thread-2933] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:00,099 DEBUG [Thread-2922] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x420f945a to 127.0.0.1:62626 2023-04-13 14:56:00,099 DEBUG [Thread-2922] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:00,101 DEBUG [Thread-2927] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70e2be04 to 127.0.0.1:62626 2023-04-13 14:56:00,101 DEBUG [Thread-2927] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:00,101 DEBUG [Thread-2931] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78592df0 to 127.0.0.1:62626 2023-04-13 14:56:00,101 DEBUG [Thread-2931] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:00,102 DEBUG [Thread-2920] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x052203cb to 127.0.0.1:62626 2023-04-13 14:56:00,102 DEBUG [Thread-2920] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:00,105 DEBUG [Thread-2935] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x680e773e to 127.0.0.1:62626 2023-04-13 14:56:00,106 DEBUG [Thread-2935] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:00,241 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/bd759a2fe28447c9bf9833bdb38b0c4b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/bd759a2fe28447c9bf9833bdb38b0c4b 2023-04-13 14:56:00,244 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/A of 46dced31cbc65aff91a95c238f6401f0 into bd759a2fe28447c9bf9833bdb38b0c4b(size=13.3 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:00,244 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:56:00,244 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/A, priority=13, startTime=1681397759785; duration=0sec 2023-04-13 14:56:00,244 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:00,497 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=889 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/4f9c34519dce4b82a788ff1e038bd847 2023-04-13 14:56:00,907 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=889 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/87337976ec06469ebf4963b2f2d20316 2023-04-13 14:56:00,910 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/3f7a7ac61e8e4514a72c89069c774194 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3f7a7ac61e8e4514a72c89069c774194 2023-04-13 14:56:00,913 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3f7a7ac61e8e4514a72c89069c774194, entries=150, sequenceid=889, filesize=11.9 K 2023-04-13 14:56:00,914 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/4f9c34519dce4b82a788ff1e038bd847 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/4f9c34519dce4b82a788ff1e038bd847 2023-04-13 14:56:00,917 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/4f9c34519dce4b82a788ff1e038bd847, entries=100, sequenceid=889, filesize=9.5 K 2023-04-13 14:56:00,917 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/87337976ec06469ebf4963b2f2d20316 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/87337976ec06469ebf4963b2f2d20316 2023-04-13 14:56:00,920 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/87337976ec06469ebf4963b2f2d20316, entries=100, sequenceid=889, filesize=9.5 K 2023-04-13 14:56:00,921 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=67.09 KB/68700 for 46dced31cbc65aff91a95c238f6401f0 in 847ms, sequenceid=889, compaction requested=true 2023-04-13 14:56:00,921 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:56:00,921 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:56:00,921 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:00,921 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:00,921 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,921 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:00,921 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:00,922 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,922 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:00,923 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:00,923 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:00,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:00,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39170f4c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:00,924 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:00,924 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:00,926 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:00,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:00,926 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:00,926 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:00,926 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:00,926 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,926 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:00,927 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:00,927 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,927 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,927 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:00,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:00,929 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:00,929 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:00,929 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:00,929 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:00,929 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:00,930 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:00,930 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:00,930 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:00,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:00,931 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:00,932 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,934 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,934 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:00,934 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:00,934 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:00,934 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:00,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:00,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:00,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:00,934 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:00,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:00,934 DEBUG [Thread-2926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:00,934 DEBUG [Thread-2926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:00,934 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:00,934 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:00,934 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:00,934 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:00,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:01,034 DEBUG [Thread-2926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:01,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:01,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:56:01,617 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 14:56:01,618 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:56:01,618 DEBUG [Thread-2918] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15880ffb to 127.0.0.1:62626 2023-04-13 14:56:01,618 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:01,618 DEBUG [Thread-2918] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:01,618 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:56:01,618 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:01,618 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:56:01,618 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:02,024 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=904 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/6987b3a1ccc74305b980443767b696f3 2023-04-13 14:56:02,432 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=904 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/489fd6a5d1804be8a7baa703a6a18a8a 2023-04-13 14:56:02,840 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=904 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/1f65e0abb31340bfa0ec74cf5801f387 2023-04-13 14:56:02,843 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/6987b3a1ccc74305b980443767b696f3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6987b3a1ccc74305b980443767b696f3 2023-04-13 14:56:02,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6987b3a1ccc74305b980443767b696f3, entries=150, sequenceid=904, filesize=11.9 K 2023-04-13 14:56:02,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/489fd6a5d1804be8a7baa703a6a18a8a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/489fd6a5d1804be8a7baa703a6a18a8a 2023-04-13 14:56:02,849 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/489fd6a5d1804be8a7baa703a6a18a8a, entries=150, sequenceid=904, filesize=11.9 K 2023-04-13 14:56:02,850 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/1f65e0abb31340bfa0ec74cf5801f387 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/1f65e0abb31340bfa0ec74cf5801f387 2023-04-13 14:56:02,852 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/1f65e0abb31340bfa0ec74cf5801f387, entries=150, sequenceid=904, filesize=11.9 K 2023-04-13 14:56:02,853 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 46dced31cbc65aff91a95c238f6401f0 in 1236ms, sequenceid=904, compaction requested=true 2023-04-13 14:56:02,853 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:56:02,853 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:02,853 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:02,853 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:56:02,853 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:56:02,853 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:02,854 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50136 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:56:02,854 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47696 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:56:02,854 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/A is initiating minor compaction (all files) 2023-04-13 14:56:02,854 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/B is initiating minor compaction (all files) 2023-04-13 14:56:02,854 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/A in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:56:02,854 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/B in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:56:02,854 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/bd759a2fe28447c9bf9833bdb38b0c4b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5c483f76131b472497d5b76db12b6423, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3f7a7ac61e8e4514a72c89069c774194, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6987b3a1ccc74305b980443767b696f3] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=49.0 K 2023-04-13 14:56:02,854 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/30db4d4d151a47788dfb96427c72e3d7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7f28c91b1cbb4ba39f11ce2529e441e9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/4f9c34519dce4b82a788ff1e038bd847, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/489fd6a5d1804be8a7baa703a6a18a8a] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=46.6 K 2023-04-13 14:56:02,855 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting bd759a2fe28447c9bf9833bdb38b0c4b, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1681397758583 2023-04-13 14:56:02,855 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 30db4d4d151a47788dfb96427c72e3d7, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1681397758583 2023-04-13 14:56:02,855 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 5c483f76131b472497d5b76db12b6423, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1681397758923 2023-04-13 14:56:02,855 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 7f28c91b1cbb4ba39f11ce2529e441e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1681397758923 2023-04-13 14:56:02,855 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 3f7a7ac61e8e4514a72c89069c774194, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=889, earliestPutTs=1681397760055 2023-04-13 14:56:02,855 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 4f9c34519dce4b82a788ff1e038bd847, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=889, earliestPutTs=1681397760055 2023-04-13 14:56:02,855 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 6987b3a1ccc74305b980443767b696f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=904, earliestPutTs=1681397760090 2023-04-13 14:56:02,855 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 489fd6a5d1804be8a7baa703a6a18a8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=904, earliestPutTs=1681397760090 2023-04-13 14:56:02,862 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#B#compaction#508 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:02,862 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#A#compaction#509 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:02,869 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/956f88ee15014c8a9a442359a76ce2b4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/956f88ee15014c8a9a442359a76ce2b4 2023-04-13 14:56:02,872 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/B of 46dced31cbc65aff91a95c238f6401f0 into 956f88ee15014c8a9a442359a76ce2b4(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:02,872 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:56:02,872 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/B, priority=12, startTime=1681397762853; duration=0sec 2023-04-13 14:56:02,872 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:02,872 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:56:02,873 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47662 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:56:02,873 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 46dced31cbc65aff91a95c238f6401f0/C is initiating minor compaction (all files) 2023-04-13 14:56:02,873 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 46dced31cbc65aff91a95c238f6401f0/C in TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:56:02,873 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/baa282af591f48cfac27ea153ffa7c7a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/2d70fcc33dfe4fc2b443002131b8a312, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/87337976ec06469ebf4963b2f2d20316, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/1f65e0abb31340bfa0ec74cf5801f387] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp, totalSize=46.5 K 2023-04-13 14:56:02,873 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting baa282af591f48cfac27ea153ffa7c7a, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=868, earliestPutTs=1681397758583 2023-04-13 14:56:02,874 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 2d70fcc33dfe4fc2b443002131b8a312, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1681397758923 2023-04-13 14:56:02,874 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 87337976ec06469ebf4963b2f2d20316, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=889, earliestPutTs=1681397760055 2023-04-13 14:56:02,874 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 1f65e0abb31340bfa0ec74cf5801f387, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=904, earliestPutTs=1681397760090 2023-04-13 14:56:02,879 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 46dced31cbc65aff91a95c238f6401f0#C#compaction#510 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:02,885 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/729d7f300e624d74a60b16cf49ff24ad as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/729d7f300e624d74a60b16cf49ff24ad 2023-04-13 14:56:02,889 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/C of 46dced31cbc65aff91a95c238f6401f0 into 729d7f300e624d74a60b16cf49ff24ad(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:02,889 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:56:02,889 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/C, priority=12, startTime=1681397762853; duration=0sec 2023-04-13 14:56:02,889 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:03,269 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/fc9cee4711f84c22975db8651ca877a5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/fc9cee4711f84c22975db8651ca877a5 2023-04-13 14:56:03,272 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 46dced31cbc65aff91a95c238f6401f0/A of 46dced31cbc65aff91a95c238f6401f0 into fc9cee4711f84c22975db8651ca877a5(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:03,272 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:56:03,272 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0., storeName=46dced31cbc65aff91a95c238f6401f0/A, priority=12, startTime=1681397762853; duration=0sec 2023-04-13 14:56:03,273 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:03,314 DEBUG [Thread-2924] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51b8df2a to 127.0.0.1:62626 2023-04-13 14:56:03,314 DEBUG [Thread-2924] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:03,362 DEBUG [Thread-2916] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37705401 to 127.0.0.1:62626 2023-04-13 14:56:03,362 DEBUG [Thread-2916] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:03,362 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 37 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 188 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 258 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 252 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 34 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(402): scanned 1683 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(403): verified 5049 rows 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(402): scanned 1678 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(403): verified 5031 rows 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(402): scanned 1675 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(403): verified 5025 rows 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(402): scanned 1676 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(403): verified 5022 rows 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(402): scanned 1685 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(403): verified 5054 rows 2023-04-13 14:56:03,363 INFO [Listener at localhost/37101] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 14:56:03,363 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a987907 to 127.0.0.1:62626 2023-04-13 14:56:03,363 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:03,365 INFO [Listener at localhost/37101] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 14:56:03,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 14:56:03,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:03,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 14:56:03,370 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397763369"}]},"ts":"1681397763369"} 2023-04-13 14:56:03,371 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 14:56:03,468 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 14:56:03,469 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=46dced31cbc65aff91a95c238f6401f0, UNASSIGN}] 2023-04-13 14:56:03,470 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=46dced31cbc65aff91a95c238f6401f0, UNASSIGN 2023-04-13 14:56:03,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 14:56:03,471 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=46dced31cbc65aff91a95c238f6401f0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:03,471 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397763471"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397763471"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397763471"}]},"ts":"1681397763471"} 2023-04-13 14:56:03,472 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:56:03,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:56:03,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 46dced31cbc65aff91a95c238f6401f0, disabling compactions & flushes 2023-04-13 14:56:03,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:56:03,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:56:03,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. after waiting 0 ms 2023-04-13 14:56:03,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:56:03,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 46dced31cbc65aff91a95c238f6401f0 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-13 14:56:03,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=A 2023-04-13 14:56:03,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:03,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=B 2023-04-13 14:56:03,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:03,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 46dced31cbc65aff91a95c238f6401f0, store=C 2023-04-13 14:56:03,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:03,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 14:56:03,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 14:56:04,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=912 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/d841146e9ef74daea38fcb79402e325b 2023-04-13 14:56:04,439 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=912 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/15fa5916aad6472da08c755aa114de6e 2023-04-13 14:56:04,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 14:56:04,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=912 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/0a5f5d7fba644def99b7a52af6c48aa2 2023-04-13 14:56:04,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/A/d841146e9ef74daea38fcb79402e325b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/d841146e9ef74daea38fcb79402e325b 2023-04-13 14:56:04,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/d841146e9ef74daea38fcb79402e325b, entries=50, sequenceid=912, filesize=7.1 K 2023-04-13 14:56:04,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/B/15fa5916aad6472da08c755aa114de6e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/15fa5916aad6472da08c755aa114de6e 2023-04-13 14:56:04,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/15fa5916aad6472da08c755aa114de6e, entries=50, sequenceid=912, filesize=7.1 K 2023-04-13 14:56:04,858 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/.tmp/C/0a5f5d7fba644def99b7a52af6c48aa2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0a5f5d7fba644def99b7a52af6c48aa2 2023-04-13 14:56:04,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0a5f5d7fba644def99b7a52af6c48aa2, entries=50, sequenceid=912, filesize=7.1 K 2023-04-13 14:56:04,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 46dced31cbc65aff91a95c238f6401f0 in 1237ms, sequenceid=912, compaction requested=false 2023-04-13 14:56:04,888 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/79768685d3344e54bd4b76c60df87be3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/205ba22091474a5eadaa1cb6dccd7cfb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/84ad7ead528240578db3249f6f7e7f15, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/055bf8ca45494580b91925853ab14c20, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/e5e8f7df9b12494e8b11a1a52a9b3112, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/585c238028fa4fcc98fda04908eee75f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5a50fa45fcd3438da1598d54151b486e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/a86331a6d532427e8f33e2b3aa8a5906, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3126bfed280a42488549fd61da9937a5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/600a1a33df314420879345947dcae407, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/2de13d159da948e5b9716d9c4ef384f2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/0c8ff876e6e444878322803ed945c98b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/74081aff99d144aa81022acb6af20ffd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6ef4273160d843249bea6c72dc7f2913, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/15698f856129465f84ca792a067f2a45, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/23c20183717b4979988eb569fe99e6f4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/86086e409f7940109715e04ed67d4e1c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/314ce864cbaa4bf39953bd2e85dafe3c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/124132879274472d8e1a95b140d95154, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/90cf1691fd49453793cb5a7884a70b3c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/c04d20c5beaf406b9254e52de81de04e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6e306cd254a94b1b9661a560a6b9efa8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3a0add52ee1746c09283d802d3715943, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8c276456052f4dd997913e95cc8ffc00, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/1579ac04717345aa801b9cf95018b41f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/4fb28703c9c145c9bd48f6fdf3a689a4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/491351f3b8dd421c83b5b56f2cfdc754, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5e6d0757994243c39fcc5b88611e95c6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8045de5a1d8f4669986b63e03e74fde6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8c157bc6f0f544f3b25bc31edcbf1f6a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/cb410ad29b014af7acf71f75b3f5768f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/900a81d8a9744abcb16aa3fcb4aa65d2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/230f0c9fdda140d9b5724c2ced264c3f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/14e144c2107c49b783de4d235940cf45, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/d89a1814c3b342e4ae5ed55d3d229a08, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/7731556ca91a4fb096547830a90e1342, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/297999a17f2f4ff8aeea4195bb902c8c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/b702741bffc0486a9ad65e4004d76633, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/0b05c80df0874bc2a02b92a5195a0d00, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/c2f1979e11ac40d8b63ebb2203d10e62, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/ddfd57599d864f0ab8d7f1ecc3db9dc7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/bd759a2fe28447c9bf9833bdb38b0c4b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5c483f76131b472497d5b76db12b6423, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3f7a7ac61e8e4514a72c89069c774194, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6987b3a1ccc74305b980443767b696f3] to archive 2023-04-13 14:56:04,890 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:56:04,891 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/79768685d3344e54bd4b76c60df87be3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/79768685d3344e54bd4b76c60df87be3 2023-04-13 14:56:04,892 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/205ba22091474a5eadaa1cb6dccd7cfb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/205ba22091474a5eadaa1cb6dccd7cfb 2023-04-13 14:56:04,893 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/84ad7ead528240578db3249f6f7e7f15 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/84ad7ead528240578db3249f6f7e7f15 2023-04-13 14:56:04,894 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/055bf8ca45494580b91925853ab14c20 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/055bf8ca45494580b91925853ab14c20 2023-04-13 14:56:04,895 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/e5e8f7df9b12494e8b11a1a52a9b3112 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/e5e8f7df9b12494e8b11a1a52a9b3112 2023-04-13 14:56:04,896 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/585c238028fa4fcc98fda04908eee75f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/585c238028fa4fcc98fda04908eee75f 2023-04-13 14:56:04,897 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5a50fa45fcd3438da1598d54151b486e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5a50fa45fcd3438da1598d54151b486e 2023-04-13 14:56:04,897 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/a86331a6d532427e8f33e2b3aa8a5906 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/a86331a6d532427e8f33e2b3aa8a5906 2023-04-13 14:56:04,898 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3126bfed280a42488549fd61da9937a5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3126bfed280a42488549fd61da9937a5 2023-04-13 14:56:04,899 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/600a1a33df314420879345947dcae407 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/600a1a33df314420879345947dcae407 2023-04-13 14:56:04,900 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/2de13d159da948e5b9716d9c4ef384f2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/2de13d159da948e5b9716d9c4ef384f2 2023-04-13 14:56:04,901 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/0c8ff876e6e444878322803ed945c98b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/0c8ff876e6e444878322803ed945c98b 2023-04-13 14:56:04,902 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/74081aff99d144aa81022acb6af20ffd to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/74081aff99d144aa81022acb6af20ffd 2023-04-13 14:56:04,902 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6ef4273160d843249bea6c72dc7f2913 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6ef4273160d843249bea6c72dc7f2913 2023-04-13 14:56:04,903 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/15698f856129465f84ca792a067f2a45 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/15698f856129465f84ca792a067f2a45 2023-04-13 14:56:04,904 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/23c20183717b4979988eb569fe99e6f4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/23c20183717b4979988eb569fe99e6f4 2023-04-13 14:56:04,905 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/86086e409f7940109715e04ed67d4e1c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/86086e409f7940109715e04ed67d4e1c 2023-04-13 14:56:04,906 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/314ce864cbaa4bf39953bd2e85dafe3c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/314ce864cbaa4bf39953bd2e85dafe3c 2023-04-13 14:56:04,907 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/124132879274472d8e1a95b140d95154 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/124132879274472d8e1a95b140d95154 2023-04-13 14:56:04,908 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/90cf1691fd49453793cb5a7884a70b3c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/90cf1691fd49453793cb5a7884a70b3c 2023-04-13 14:56:04,908 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/c04d20c5beaf406b9254e52de81de04e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/c04d20c5beaf406b9254e52de81de04e 2023-04-13 14:56:04,909 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6e306cd254a94b1b9661a560a6b9efa8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6e306cd254a94b1b9661a560a6b9efa8 2023-04-13 14:56:04,910 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3a0add52ee1746c09283d802d3715943 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3a0add52ee1746c09283d802d3715943 2023-04-13 14:56:04,911 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8c276456052f4dd997913e95cc8ffc00 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8c276456052f4dd997913e95cc8ffc00 2023-04-13 14:56:04,911 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/1579ac04717345aa801b9cf95018b41f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/1579ac04717345aa801b9cf95018b41f 2023-04-13 14:56:04,912 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/4fb28703c9c145c9bd48f6fdf3a689a4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/4fb28703c9c145c9bd48f6fdf3a689a4 2023-04-13 14:56:04,913 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/491351f3b8dd421c83b5b56f2cfdc754 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/491351f3b8dd421c83b5b56f2cfdc754 2023-04-13 14:56:04,914 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5e6d0757994243c39fcc5b88611e95c6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5e6d0757994243c39fcc5b88611e95c6 2023-04-13 14:56:04,914 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8045de5a1d8f4669986b63e03e74fde6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8045de5a1d8f4669986b63e03e74fde6 2023-04-13 14:56:04,915 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8c157bc6f0f544f3b25bc31edcbf1f6a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/8c157bc6f0f544f3b25bc31edcbf1f6a 2023-04-13 14:56:04,916 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/cb410ad29b014af7acf71f75b3f5768f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/cb410ad29b014af7acf71f75b3f5768f 2023-04-13 14:56:04,917 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/900a81d8a9744abcb16aa3fcb4aa65d2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/900a81d8a9744abcb16aa3fcb4aa65d2 2023-04-13 14:56:04,918 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/230f0c9fdda140d9b5724c2ced264c3f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/230f0c9fdda140d9b5724c2ced264c3f 2023-04-13 14:56:04,919 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/14e144c2107c49b783de4d235940cf45 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/14e144c2107c49b783de4d235940cf45 2023-04-13 14:56:04,920 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/d89a1814c3b342e4ae5ed55d3d229a08 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/d89a1814c3b342e4ae5ed55d3d229a08 2023-04-13 14:56:04,921 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/7731556ca91a4fb096547830a90e1342 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/7731556ca91a4fb096547830a90e1342 2023-04-13 14:56:04,922 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/297999a17f2f4ff8aeea4195bb902c8c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/297999a17f2f4ff8aeea4195bb902c8c 2023-04-13 14:56:04,923 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/b702741bffc0486a9ad65e4004d76633 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/b702741bffc0486a9ad65e4004d76633 2023-04-13 14:56:04,924 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/0b05c80df0874bc2a02b92a5195a0d00 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/0b05c80df0874bc2a02b92a5195a0d00 2023-04-13 14:56:04,925 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/c2f1979e11ac40d8b63ebb2203d10e62 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/c2f1979e11ac40d8b63ebb2203d10e62 2023-04-13 14:56:04,926 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/ddfd57599d864f0ab8d7f1ecc3db9dc7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/ddfd57599d864f0ab8d7f1ecc3db9dc7 2023-04-13 14:56:04,927 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/bd759a2fe28447c9bf9833bdb38b0c4b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/bd759a2fe28447c9bf9833bdb38b0c4b 2023-04-13 14:56:04,928 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5c483f76131b472497d5b76db12b6423 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/5c483f76131b472497d5b76db12b6423 2023-04-13 14:56:04,929 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3f7a7ac61e8e4514a72c89069c774194 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/3f7a7ac61e8e4514a72c89069c774194 2023-04-13 14:56:04,930 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6987b3a1ccc74305b980443767b696f3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/6987b3a1ccc74305b980443767b696f3 2023-04-13 14:56:04,933 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/76a10b5424854f10a7f9ca1ae4ae4205, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/03bdb8788a014719a5718b5de4af6507, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/37c2b33931054730926f504472264ded, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/d851bfde2c3241919acf87f42d10f77b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/bfd1c8d96185437bb58ba82b8e7ffb3b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/f8711b7405e34711b903822b6166f5a8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/399fe877f3e6444797cf534ad0d32fef, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/fcae84eda8a24637abae383d6180fd6d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/e743378e48284f7aa157f646c4bf2f0a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/77fcf1da87ce4c15980ea615159029e4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/444e1242a6e8463095279c8176faad34, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/0d5c2e91523e42fd80da88ac59f6a834, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/9821c823c49443b0acc82b8fc43b0db3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/73486f2ddd484217b9aaf2d302a01edb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/c73d5550ae984c21964ee4f4d6ff5e44, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/344e4422efba4f6e99b1f81584d0bef8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/b9203e949afe47f5bbe7e3224f6d21b7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/aaf736ae904d4be0aecac655f475e990, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/5e6e842785d54559975633a4734aaadc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/372adc5de5444750959402d2bafd9ffb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/4689054a16f946f4972142e1be69b583, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7469c1ee554d404eaba2844e7dde0199, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/31e70373df554d399a0d41047e54ceab, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/450c1bedd67a400d9cd301f9bb97e1a4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/14fa4665da8c42a6ac887e42a591fc88, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/26f60155853a413bbbd1c3619b35408c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/593d619df2bb4af18026c7695cdcbb37, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6bff03f298474964ae76f75f96c71d47, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/0ac2681c084d459bb73b0e9a6877aabd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/1fbe698286a74c7bbaf8bd69d7d00d05, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/2a24049fed614357b971eea720a9eba6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7d64f8b4b3894175934ed42fdcab128f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/9eb3d99eddc14750a1c24467e4d7dd32, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6617fdf157ce440e9af20eb27db6b0e3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/ceb6c75ea7c1416da590366316427367, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/29906e6dd5ae4ae58312339db959be82, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6ad64fd6298f4784aa4b0dc6e8b2b282, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/98d9c512e2f94c8aadfa47ccfc56a298, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/fef5b963d465407ca7d9804153e46701, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/32b7aa109f294fc5a704e1aba1315edb, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/30db4d4d151a47788dfb96427c72e3d7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/ff78bad9234548bfa87f061f8b1b3089, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7f28c91b1cbb4ba39f11ce2529e441e9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/4f9c34519dce4b82a788ff1e038bd847, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/489fd6a5d1804be8a7baa703a6a18a8a] to archive 2023-04-13 14:56:04,934 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:56:04,935 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/76a10b5424854f10a7f9ca1ae4ae4205 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/76a10b5424854f10a7f9ca1ae4ae4205 2023-04-13 14:56:04,936 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/03bdb8788a014719a5718b5de4af6507 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/03bdb8788a014719a5718b5de4af6507 2023-04-13 14:56:04,937 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/37c2b33931054730926f504472264ded to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/37c2b33931054730926f504472264ded 2023-04-13 14:56:04,938 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/d851bfde2c3241919acf87f42d10f77b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/d851bfde2c3241919acf87f42d10f77b 2023-04-13 14:56:04,938 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/bfd1c8d96185437bb58ba82b8e7ffb3b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/bfd1c8d96185437bb58ba82b8e7ffb3b 2023-04-13 14:56:04,939 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/f8711b7405e34711b903822b6166f5a8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/f8711b7405e34711b903822b6166f5a8 2023-04-13 14:56:04,940 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/399fe877f3e6444797cf534ad0d32fef to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/399fe877f3e6444797cf534ad0d32fef 2023-04-13 14:56:04,941 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/fcae84eda8a24637abae383d6180fd6d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/fcae84eda8a24637abae383d6180fd6d 2023-04-13 14:56:04,942 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/e743378e48284f7aa157f646c4bf2f0a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/e743378e48284f7aa157f646c4bf2f0a 2023-04-13 14:56:04,943 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/77fcf1da87ce4c15980ea615159029e4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/77fcf1da87ce4c15980ea615159029e4 2023-04-13 14:56:04,944 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/444e1242a6e8463095279c8176faad34 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/444e1242a6e8463095279c8176faad34 2023-04-13 14:56:04,945 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/0d5c2e91523e42fd80da88ac59f6a834 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/0d5c2e91523e42fd80da88ac59f6a834 2023-04-13 14:56:04,946 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/9821c823c49443b0acc82b8fc43b0db3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/9821c823c49443b0acc82b8fc43b0db3 2023-04-13 14:56:04,947 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/73486f2ddd484217b9aaf2d302a01edb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/73486f2ddd484217b9aaf2d302a01edb 2023-04-13 14:56:04,948 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/c73d5550ae984c21964ee4f4d6ff5e44 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/c73d5550ae984c21964ee4f4d6ff5e44 2023-04-13 14:56:04,949 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/344e4422efba4f6e99b1f81584d0bef8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/344e4422efba4f6e99b1f81584d0bef8 2023-04-13 14:56:04,950 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/b9203e949afe47f5bbe7e3224f6d21b7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/b9203e949afe47f5bbe7e3224f6d21b7 2023-04-13 14:56:04,952 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/aaf736ae904d4be0aecac655f475e990 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/aaf736ae904d4be0aecac655f475e990 2023-04-13 14:56:04,953 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/5e6e842785d54559975633a4734aaadc to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/5e6e842785d54559975633a4734aaadc 2023-04-13 14:56:04,954 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/372adc5de5444750959402d2bafd9ffb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/372adc5de5444750959402d2bafd9ffb 2023-04-13 14:56:04,955 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/4689054a16f946f4972142e1be69b583 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/4689054a16f946f4972142e1be69b583 2023-04-13 14:56:04,956 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7469c1ee554d404eaba2844e7dde0199 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7469c1ee554d404eaba2844e7dde0199 2023-04-13 14:56:04,957 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/31e70373df554d399a0d41047e54ceab to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/31e70373df554d399a0d41047e54ceab 2023-04-13 14:56:04,958 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/450c1bedd67a400d9cd301f9bb97e1a4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/450c1bedd67a400d9cd301f9bb97e1a4 2023-04-13 14:56:04,959 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/14fa4665da8c42a6ac887e42a591fc88 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/14fa4665da8c42a6ac887e42a591fc88 2023-04-13 14:56:04,960 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/26f60155853a413bbbd1c3619b35408c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/26f60155853a413bbbd1c3619b35408c 2023-04-13 14:56:04,960 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/593d619df2bb4af18026c7695cdcbb37 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/593d619df2bb4af18026c7695cdcbb37 2023-04-13 14:56:04,961 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6bff03f298474964ae76f75f96c71d47 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6bff03f298474964ae76f75f96c71d47 2023-04-13 14:56:04,962 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/0ac2681c084d459bb73b0e9a6877aabd to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/0ac2681c084d459bb73b0e9a6877aabd 2023-04-13 14:56:04,964 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/1fbe698286a74c7bbaf8bd69d7d00d05 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/1fbe698286a74c7bbaf8bd69d7d00d05 2023-04-13 14:56:04,965 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/2a24049fed614357b971eea720a9eba6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/2a24049fed614357b971eea720a9eba6 2023-04-13 14:56:04,966 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7d64f8b4b3894175934ed42fdcab128f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7d64f8b4b3894175934ed42fdcab128f 2023-04-13 14:56:04,967 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/9eb3d99eddc14750a1c24467e4d7dd32 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/9eb3d99eddc14750a1c24467e4d7dd32 2023-04-13 14:56:04,968 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6617fdf157ce440e9af20eb27db6b0e3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6617fdf157ce440e9af20eb27db6b0e3 2023-04-13 14:56:04,969 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/ceb6c75ea7c1416da590366316427367 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/ceb6c75ea7c1416da590366316427367 2023-04-13 14:56:04,970 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/29906e6dd5ae4ae58312339db959be82 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/29906e6dd5ae4ae58312339db959be82 2023-04-13 14:56:04,971 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6ad64fd6298f4784aa4b0dc6e8b2b282 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/6ad64fd6298f4784aa4b0dc6e8b2b282 2023-04-13 14:56:04,972 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/98d9c512e2f94c8aadfa47ccfc56a298 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/98d9c512e2f94c8aadfa47ccfc56a298 2023-04-13 14:56:04,973 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/fef5b963d465407ca7d9804153e46701 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/fef5b963d465407ca7d9804153e46701 2023-04-13 14:56:04,974 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/32b7aa109f294fc5a704e1aba1315edb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/32b7aa109f294fc5a704e1aba1315edb 2023-04-13 14:56:04,974 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/30db4d4d151a47788dfb96427c72e3d7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/30db4d4d151a47788dfb96427c72e3d7 2023-04-13 14:56:04,975 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/ff78bad9234548bfa87f061f8b1b3089 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/ff78bad9234548bfa87f061f8b1b3089 2023-04-13 14:56:04,976 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7f28c91b1cbb4ba39f11ce2529e441e9 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/7f28c91b1cbb4ba39f11ce2529e441e9 2023-04-13 14:56:04,977 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/4f9c34519dce4b82a788ff1e038bd847 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/4f9c34519dce4b82a788ff1e038bd847 2023-04-13 14:56:04,978 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/489fd6a5d1804be8a7baa703a6a18a8a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/489fd6a5d1804be8a7baa703a6a18a8a 2023-04-13 14:56:04,981 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/aef8f87559cc47e1ba2634054d52804b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/207be4e558a846ad92036e58d408fb5d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/07ec740fa75d4a1eaa6c23f480676425, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c3dfcf58eb28489290fb127909aa693f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/d37649126e534a1796f2c5604c03689f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/faa34179a3fc47fdb1f87131b5b23ae7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ffb0506015714422a8fdb4ac052a5b08, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/de2c94ea324f4b3c8d45cd8368f0529b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/6b5800b6807b4198a8cb1c833fdd1789, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/7e6d971a60fd47efb47db488de0cc916, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5ba3b7f326584cdbae6169eaffe88947, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/1c90f2f5cffe427f87dc531c0e17c3a8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0790b2dec1024044a06496a0e1d76f06, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5d5c75c4d9764384865107dbf47e35f2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/15a71c86b5474276a8277d2dd9b86175, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/abf56a2b3ce54954a4497e656c54c875, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/dc73ee5beb814bf6a6a0930de2665206, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c9a38a37d0034bfb9dc1d944bc9114c6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/9df4170b7dcb423ba1b0a7867ca621ee, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c0470cadaea943d6a66a01eb908d101f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/83203eb5194b4be78a8868b20eeb47dd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ff1e9124f28040da8000ede27793c067, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ff48d9e4bf984ec1b24573da0eb95f4b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/f3a1536d2aac415faa8f5ef851a2cfaf, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ec3189d2c1574361a2c7b9553b39aeef, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/566509147a044f17ae2e73610c32811e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/d7426f1057dd47ec8c442e5604dcaf68, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0aab40f0fcfe47e7809938bd3f571be8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/e090a6ca889a4b76be6708d127dd8fbc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/8277414a0932446fb476434c321cd8a3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/3f6cf16772984cebaa0352bc296d6bd8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/03939d8e73714b2686f1727c6e829eda, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/fb237007775e4e96ae3ae6e114ca45f0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/b38ee05663ae4e9ba05f3366d5c5ae28, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/71cb2a6865684d8d96880f1db05e16af, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/8ebafa09f47d47609286477b281803f0, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/247d76ef8a1845889f7e51397e782212, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/a468ae2d57514cd7bde4852ef1b295fd, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5ddc4f696ebc4fd9abbd17d43ce7b0a8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/baa282af591f48cfac27ea153ffa7c7a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/7286a33650874915be4432369df69018, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/2d70fcc33dfe4fc2b443002131b8a312, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/87337976ec06469ebf4963b2f2d20316, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/1f65e0abb31340bfa0ec74cf5801f387] to archive 2023-04-13 14:56:04,982 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:56:04,983 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/aef8f87559cc47e1ba2634054d52804b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/aef8f87559cc47e1ba2634054d52804b 2023-04-13 14:56:04,983 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/207be4e558a846ad92036e58d408fb5d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/207be4e558a846ad92036e58d408fb5d 2023-04-13 14:56:04,984 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/07ec740fa75d4a1eaa6c23f480676425 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/07ec740fa75d4a1eaa6c23f480676425 2023-04-13 14:56:04,985 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c3dfcf58eb28489290fb127909aa693f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c3dfcf58eb28489290fb127909aa693f 2023-04-13 14:56:04,986 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/d37649126e534a1796f2c5604c03689f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/d37649126e534a1796f2c5604c03689f 2023-04-13 14:56:04,987 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/faa34179a3fc47fdb1f87131b5b23ae7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/faa34179a3fc47fdb1f87131b5b23ae7 2023-04-13 14:56:04,988 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ffb0506015714422a8fdb4ac052a5b08 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ffb0506015714422a8fdb4ac052a5b08 2023-04-13 14:56:04,989 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/de2c94ea324f4b3c8d45cd8368f0529b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/de2c94ea324f4b3c8d45cd8368f0529b 2023-04-13 14:56:04,990 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/6b5800b6807b4198a8cb1c833fdd1789 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/6b5800b6807b4198a8cb1c833fdd1789 2023-04-13 14:56:04,990 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/7e6d971a60fd47efb47db488de0cc916 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/7e6d971a60fd47efb47db488de0cc916 2023-04-13 14:56:04,991 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5ba3b7f326584cdbae6169eaffe88947 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5ba3b7f326584cdbae6169eaffe88947 2023-04-13 14:56:04,992 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/1c90f2f5cffe427f87dc531c0e17c3a8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/1c90f2f5cffe427f87dc531c0e17c3a8 2023-04-13 14:56:04,993 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0790b2dec1024044a06496a0e1d76f06 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0790b2dec1024044a06496a0e1d76f06 2023-04-13 14:56:04,994 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5d5c75c4d9764384865107dbf47e35f2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5d5c75c4d9764384865107dbf47e35f2 2023-04-13 14:56:04,995 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/15a71c86b5474276a8277d2dd9b86175 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/15a71c86b5474276a8277d2dd9b86175 2023-04-13 14:56:04,995 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/abf56a2b3ce54954a4497e656c54c875 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/abf56a2b3ce54954a4497e656c54c875 2023-04-13 14:56:04,996 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/dc73ee5beb814bf6a6a0930de2665206 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/dc73ee5beb814bf6a6a0930de2665206 2023-04-13 14:56:04,997 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c9a38a37d0034bfb9dc1d944bc9114c6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c9a38a37d0034bfb9dc1d944bc9114c6 2023-04-13 14:56:04,998 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/9df4170b7dcb423ba1b0a7867ca621ee to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/9df4170b7dcb423ba1b0a7867ca621ee 2023-04-13 14:56:04,999 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c0470cadaea943d6a66a01eb908d101f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/c0470cadaea943d6a66a01eb908d101f 2023-04-13 14:56:04,999 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/83203eb5194b4be78a8868b20eeb47dd to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/83203eb5194b4be78a8868b20eeb47dd 2023-04-13 14:56:05,000 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ff1e9124f28040da8000ede27793c067 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ff1e9124f28040da8000ede27793c067 2023-04-13 14:56:05,001 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ff48d9e4bf984ec1b24573da0eb95f4b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ff48d9e4bf984ec1b24573da0eb95f4b 2023-04-13 14:56:05,002 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/f3a1536d2aac415faa8f5ef851a2cfaf to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/f3a1536d2aac415faa8f5ef851a2cfaf 2023-04-13 14:56:05,003 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ec3189d2c1574361a2c7b9553b39aeef to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/ec3189d2c1574361a2c7b9553b39aeef 2023-04-13 14:56:05,004 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/566509147a044f17ae2e73610c32811e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/566509147a044f17ae2e73610c32811e 2023-04-13 14:56:05,005 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/d7426f1057dd47ec8c442e5604dcaf68 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/d7426f1057dd47ec8c442e5604dcaf68 2023-04-13 14:56:05,006 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0aab40f0fcfe47e7809938bd3f571be8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0aab40f0fcfe47e7809938bd3f571be8 2023-04-13 14:56:05,007 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/e090a6ca889a4b76be6708d127dd8fbc to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/e090a6ca889a4b76be6708d127dd8fbc 2023-04-13 14:56:05,008 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/8277414a0932446fb476434c321cd8a3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/8277414a0932446fb476434c321cd8a3 2023-04-13 14:56:05,009 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/3f6cf16772984cebaa0352bc296d6bd8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/3f6cf16772984cebaa0352bc296d6bd8 2023-04-13 14:56:05,010 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/03939d8e73714b2686f1727c6e829eda to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/03939d8e73714b2686f1727c6e829eda 2023-04-13 14:56:05,011 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/fb237007775e4e96ae3ae6e114ca45f0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/fb237007775e4e96ae3ae6e114ca45f0 2023-04-13 14:56:05,012 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/b38ee05663ae4e9ba05f3366d5c5ae28 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/b38ee05663ae4e9ba05f3366d5c5ae28 2023-04-13 14:56:05,013 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/71cb2a6865684d8d96880f1db05e16af to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/71cb2a6865684d8d96880f1db05e16af 2023-04-13 14:56:05,014 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/8ebafa09f47d47609286477b281803f0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/8ebafa09f47d47609286477b281803f0 2023-04-13 14:56:05,015 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/247d76ef8a1845889f7e51397e782212 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/247d76ef8a1845889f7e51397e782212 2023-04-13 14:56:05,016 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/a468ae2d57514cd7bde4852ef1b295fd to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/a468ae2d57514cd7bde4852ef1b295fd 2023-04-13 14:56:05,017 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5ddc4f696ebc4fd9abbd17d43ce7b0a8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/5ddc4f696ebc4fd9abbd17d43ce7b0a8 2023-04-13 14:56:05,018 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/baa282af591f48cfac27ea153ffa7c7a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/baa282af591f48cfac27ea153ffa7c7a 2023-04-13 14:56:05,019 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/7286a33650874915be4432369df69018 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/7286a33650874915be4432369df69018 2023-04-13 14:56:05,020 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/2d70fcc33dfe4fc2b443002131b8a312 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/2d70fcc33dfe4fc2b443002131b8a312 2023-04-13 14:56:05,021 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/87337976ec06469ebf4963b2f2d20316 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/87337976ec06469ebf4963b2f2d20316 2023-04-13 14:56:05,022 DEBUG [StoreCloser-TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/1f65e0abb31340bfa0ec74cf5801f387 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/1f65e0abb31340bfa0ec74cf5801f387 2023-04-13 14:56:05,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/recovered.edits/915.seqid, newMaxSeqId=915, maxSeqId=1 2023-04-13 14:56:05,027 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0. 2023-04-13 14:56:05,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 46dced31cbc65aff91a95c238f6401f0: 2023-04-13 14:56:05,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:56:05,028 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=46dced31cbc65aff91a95c238f6401f0, regionState=CLOSED 2023-04-13 14:56:05,029 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397765028"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397765028"}]},"ts":"1681397765028"} 2023-04-13 14:56:05,031 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-13 14:56:05,031 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 46dced31cbc65aff91a95c238f6401f0, server=jenkins-hbase4.apache.org,43875,1681397617638 in 1.5580 sec 2023-04-13 14:56:05,032 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-13 14:56:05,032 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=46dced31cbc65aff91a95c238f6401f0, UNASSIGN in 1.5620 sec 2023-04-13 14:56:05,033 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397765032"}]},"ts":"1681397765032"} 2023-04-13 14:56:05,033 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 14:56:05,035 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 14:56:05,036 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6700 sec 2023-04-13 14:56:05,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 14:56:05,474 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-13 14:56:05,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 14:56:05,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:05,477 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:05,477 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:05,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-13 14:56:05,482 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:56:05,484 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/recovered.edits] 2023-04-13 14:56:05,487 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/d841146e9ef74daea38fcb79402e325b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/d841146e9ef74daea38fcb79402e325b 2023-04-13 14:56:05,488 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/fc9cee4711f84c22975db8651ca877a5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/A/fc9cee4711f84c22975db8651ca877a5 2023-04-13 14:56:05,490 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/15fa5916aad6472da08c755aa114de6e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/15fa5916aad6472da08c755aa114de6e 2023-04-13 14:56:05,491 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/956f88ee15014c8a9a442359a76ce2b4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/B/956f88ee15014c8a9a442359a76ce2b4 2023-04-13 14:56:05,493 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0a5f5d7fba644def99b7a52af6c48aa2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/0a5f5d7fba644def99b7a52af6c48aa2 2023-04-13 14:56:05,494 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/729d7f300e624d74a60b16cf49ff24ad to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/C/729d7f300e624d74a60b16cf49ff24ad 2023-04-13 14:56:05,496 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/recovered.edits/915.seqid to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0/recovered.edits/915.seqid 2023-04-13 14:56:05,497 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/46dced31cbc65aff91a95c238f6401f0 2023-04-13 14:56:05,497 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 14:56:05,499 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:05,500 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 14:56:05,502 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 14:56:05,502 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:05,503 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 14:56:05,503 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681397765503"}]},"ts":"9223372036854775807"} 2023-04-13 14:56:05,504 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 14:56:05,504 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 46dced31cbc65aff91a95c238f6401f0, NAME => 'TestAcidGuarantees,,1681397737897.46dced31cbc65aff91a95c238f6401f0.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 14:56:05,504 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 14:56:05,504 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681397765504"}]},"ts":"9223372036854775807"} 2023-04-13 14:56:05,505 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 14:56:05,507 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:05,508 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 32 msec 2023-04-13 14:56:05,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-13 14:56:05,578 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-13 14:56:05,590 INFO [Listener at localhost/37101] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229 (was 229), OpenFileDescriptor=592 (was 538) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=448 (was 421) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=1141 (was 662) - AvailableMemoryMB LEAK? - 2023-04-13 14:56:05,602 INFO [Listener at localhost/37101] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229, OpenFileDescriptor=592, MaxFileDescriptor=60000, SystemLoadAverage=448, ProcessCount=169, AvailableMemoryMB=1141 2023-04-13 14:56:05,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 14:56:05,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 14:56:05,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:05,606 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 14:56:05,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-13 14:56:05,607 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 14:56:05,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 14:56:05,608 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:05,608 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5 empty. 2023-04-13 14:56:05,609 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:05,609 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 14:56:05,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 14:56:05,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 14:56:06,015 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 14:56:06,017 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7f6fba518b666b8072f5bc2fa5e4a5d5, NAME => 'TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp 2023-04-13 14:56:06,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 14:56:06,422 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:56:06,422 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 7f6fba518b666b8072f5bc2fa5e4a5d5, disabling compactions & flushes 2023-04-13 14:56:06,422 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:06,422 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:06,422 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. after waiting 0 ms 2023-04-13 14:56:06,422 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:06,422 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:06,422 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:06,424 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 14:56:06,425 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397766425"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397766425"}]},"ts":"1681397766425"} 2023-04-13 14:56:06,426 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 14:56:06,426 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 14:56:06,426 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397766426"}]},"ts":"1681397766426"} 2023-04-13 14:56:06,427 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 14:56:06,429 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f6fba518b666b8072f5bc2fa5e4a5d5, ASSIGN}] 2023-04-13 14:56:06,431 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f6fba518b666b8072f5bc2fa5e4a5d5, ASSIGN 2023-04-13 14:56:06,431 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f6fba518b666b8072f5bc2fa5e4a5d5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43875,1681397617638; forceNewPlan=false, retain=false 2023-04-13 14:56:06,583 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=7f6fba518b666b8072f5bc2fa5e4a5d5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:06,583 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397766583"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397766583"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397766583"}]},"ts":"1681397766583"} 2023-04-13 14:56:06,584 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:56:06,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 14:56:06,740 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:06,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7f6fba518b666b8072f5bc2fa5e4a5d5, NAME => 'TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.', STARTKEY => '', ENDKEY => ''} 2023-04-13 14:56:06,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:06,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:56:06,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:06,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:06,742 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:06,744 DEBUG [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A 2023-04-13 14:56:06,744 DEBUG [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A 2023-04-13 14:56:06,744 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:56:06,744 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7f6fba518b666b8072f5bc2fa5e4a5d5 columnFamilyName A 2023-04-13 14:56:06,745 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.HStore(310): Store=7f6fba518b666b8072f5bc2fa5e4a5d5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:56:06,745 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:06,746 DEBUG [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B 2023-04-13 14:56:06,747 DEBUG [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B 2023-04-13 14:56:06,747 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:56:06,747 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7f6fba518b666b8072f5bc2fa5e4a5d5 columnFamilyName B 2023-04-13 14:56:06,748 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.HStore(310): Store=7f6fba518b666b8072f5bc2fa5e4a5d5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:56:06,748 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:06,749 DEBUG [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C 2023-04-13 14:56:06,749 DEBUG [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C 2023-04-13 14:56:06,749 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:56:06,749 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7f6fba518b666b8072f5bc2fa5e4a5d5 columnFamilyName C 2023-04-13 14:56:06,750 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.HStore(310): Store=7f6fba518b666b8072f5bc2fa5e4a5d5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:56:06,750 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:06,750 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:06,751 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:06,752 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 14:56:06,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:06,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 14:56:06,755 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7f6fba518b666b8072f5bc2fa5e4a5d5; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60695836, jitterRate=-0.09556156396865845}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 14:56:06,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:06,756 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., pid=56, masterSystemTime=1681397766735 2023-04-13 14:56:06,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:06,757 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:06,757 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=7f6fba518b666b8072f5bc2fa5e4a5d5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:06,758 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397766757"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681397766757"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681397766757"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681397766757"}]},"ts":"1681397766757"} 2023-04-13 14:56:06,759 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-13 14:56:06,759 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 in 174 msec 2023-04-13 14:56:06,761 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-13 14:56:06,761 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f6fba518b666b8072f5bc2fa5e4a5d5, ASSIGN in 330 msec 2023-04-13 14:56:06,761 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 14:56:06,761 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397766761"}]},"ts":"1681397766761"} 2023-04-13 14:56:06,762 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 14:56:06,765 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 14:56:06,765 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-04-13 14:56:07,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 14:56:07,711 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-13 14:56:07,712 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x470742aa to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:56:07,717 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ced4ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:56:07,718 DEBUG [Listener at localhost/37101] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:56:07,719 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38360, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:56:07,720 DEBUG [Listener at localhost/37101] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 14:56:07,723 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54476, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 14:56:07,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 14:56:07,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 14:56:07,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:08,139 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 14:56:08,140 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 14:56:08,140 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 14:56:08,142 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-13 14:56:08,144 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f6fba518b666b8072f5bc2fa5e4a5d5, REOPEN/MOVE}] 2023-04-13 14:56:08,145 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f6fba518b666b8072f5bc2fa5e4a5d5, REOPEN/MOVE 2023-04-13 14:56:08,145 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=7f6fba518b666b8072f5bc2fa5e4a5d5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,145 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397768145"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397768145"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397768145"}]},"ts":"1681397768145"} 2023-04-13 14:56:08,146 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:56:08,298 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:08,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7f6fba518b666b8072f5bc2fa5e4a5d5, disabling compactions & flushes 2023-04-13 14:56:08,299 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:08,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:08,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. after waiting 0 ms 2023-04-13 14:56:08,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:08,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-13 14:56:08,302 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:08,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:08,302 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 7f6fba518b666b8072f5bc2fa5e4a5d5 to self. 2023-04-13 14:56:08,304 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:08,304 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=7f6fba518b666b8072f5bc2fa5e4a5d5, regionState=CLOSED 2023-04-13 14:56:08,304 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397768304"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397768304"}]},"ts":"1681397768304"} 2023-04-13 14:56:08,306 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-13 14:56:08,306 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 in 159 msec 2023-04-13 14:56:08,307 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f6fba518b666b8072f5bc2fa5e4a5d5, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,43875,1681397617638; forceNewPlan=false, retain=true 2023-04-13 14:56:08,457 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=7f6fba518b666b8072f5bc2fa5e4a5d5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,457 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397768457"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397768457"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397768457"}]},"ts":"1681397768457"} 2023-04-13 14:56:08,458 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:56:08,612 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:08,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7f6fba518b666b8072f5bc2fa5e4a5d5, NAME => 'TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.', STARTKEY => '', ENDKEY => ''} 2023-04-13 14:56:08,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:08,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 14:56:08,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:08,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:08,614 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:08,615 DEBUG [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A 2023-04-13 14:56:08,615 DEBUG [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A 2023-04-13 14:56:08,615 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:56:08,615 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7f6fba518b666b8072f5bc2fa5e4a5d5 columnFamilyName A 2023-04-13 14:56:08,616 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.HStore(310): Store=7f6fba518b666b8072f5bc2fa5e4a5d5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:56:08,616 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:08,617 DEBUG [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B 2023-04-13 14:56:08,617 DEBUG [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B 2023-04-13 14:56:08,617 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:56:08,617 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7f6fba518b666b8072f5bc2fa5e4a5d5 columnFamilyName B 2023-04-13 14:56:08,618 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.HStore(310): Store=7f6fba518b666b8072f5bc2fa5e4a5d5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:56:08,618 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:08,618 DEBUG [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C 2023-04-13 14:56:08,618 DEBUG [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C 2023-04-13 14:56:08,618 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 14:56:08,619 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7f6fba518b666b8072f5bc2fa5e4a5d5 columnFamilyName C 2023-04-13 14:56:08,619 INFO [StoreOpener-7f6fba518b666b8072f5bc2fa5e4a5d5-1] regionserver.HStore(310): Store=7f6fba518b666b8072f5bc2fa5e4a5d5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 14:56:08,619 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:08,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:08,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:08,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 14:56:08,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:08,623 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7f6fba518b666b8072f5bc2fa5e4a5d5; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65001098, jitterRate=-0.0314081609249115}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 14:56:08,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:08,624 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., pid=61, masterSystemTime=1681397768610 2023-04-13 14:56:08,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:08,625 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:08,625 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=7f6fba518b666b8072f5bc2fa5e4a5d5, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,625 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397768625"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681397768625"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681397768625"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681397768625"}]},"ts":"1681397768625"} 2023-04-13 14:56:08,627 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-13 14:56:08,627 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 in 168 msec 2023-04-13 14:56:08,628 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-13 14:56:08,628 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f6fba518b666b8072f5bc2fa5e4a5d5, REOPEN/MOVE in 483 msec 2023-04-13 14:56:08,630 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-13 14:56:08,630 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 487 msec 2023-04-13 14:56:08,631 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 905 msec 2023-04-13 14:56:08,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-13 14:56:08,633 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x1e3503b4 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:56:08,638 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63da10ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:56:08,638 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x6f06dd50 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:56:08,642 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59185ba7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:56:08,642 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e8654 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:56:08,649 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62b3caa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:56:08,650 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x4db22237 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:56:08,653 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66b5b1f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:56:08,653 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x08b57aae to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:56:08,657 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@544a4215, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:56:08,657 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x25baee3e to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:56:08,663 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23e65a76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:56:08,663 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x2c53bf96 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:56:08,671 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e6a6f3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:56:08,671 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x3b59bdcf to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:56:08,677 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a470a3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:56:08,677 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x3d765193 to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:56:08,683 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bb72437, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:56:08,683 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(139): Connect 0x32bf16cd to 127.0.0.1:62626 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 14:56:08,690 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dca23c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 14:56:08,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:08,700 DEBUG [hconnection-0x5920204f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:56:08,700 DEBUG [hconnection-0x7bf55d29-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:56:08,701 DEBUG [hconnection-0x36ff966e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:56:08,702 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38372, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:56:08,704 DEBUG [hconnection-0x23d90ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:56:08,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:08,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:08,708 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:08,708 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:08,708 DEBUG [hconnection-0x1c6de5f9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:56:08,708 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:08,708 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:08,708 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,708 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:08,709 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:08,709 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:08,710 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,710 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:08,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:08,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:08,712 DEBUG [hconnection-0x6dd3700d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:56:08,712 DEBUG [hconnection-0x76384951-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:56:08,713 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38410, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:56:08,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:08,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:08,716 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:08,718 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:08,718 DEBUG [hconnection-0x37523742-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:56:08,718 DEBUG [hconnection-0x218b55b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:56:08,718 DEBUG [hconnection-0x2cc0fd84-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 14:56:08,720 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38370, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:56:08,718 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:08,720 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38362, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:56:08,721 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38388, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:56:08,721 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38394, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:56:08,721 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:08,721 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38426, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:56:08,721 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:08,721 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:08,721 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:08,725 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:08,726 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:08,726 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:08,726 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:08,726 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,726 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38434, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:56:08,726 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:08,726 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38442, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:56:08,727 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:08,728 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38448, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 14:56:08,731 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,731 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:08,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:08,731 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,731 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:08,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:08,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:08,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:08,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c4d551[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:08,734 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:08,734 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,735 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,735 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:08,735 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,735 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:08,735 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:08,735 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:08,737 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,737 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:08,737 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:08,737 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:08,738 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:56:08,739 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:08,739 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:08,739 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:08,739 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:08,739 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:08,739 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:08,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:08,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:08,746 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:08,747 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:08,747 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:08,747 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:08,747 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:08,748 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,748 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,748 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:08,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:08,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:08,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,749 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,750 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:08,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:08,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:08,752 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:08,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:08,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:08,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:08,752 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:08,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:08,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:08,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,752 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:08,752 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:08,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:08,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:08,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:08,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397768753, max timestamp=1681397768757Num uniques 859; 2023-04-13 14:56:08,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397768753, max timestamp=1681397768757Num uniques 859; 2023-04-13 14:56:08,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397768753, max timestamp=1681397768757Num uniques 859; 2023-04-13 14:56:08,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:08,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:08,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:08,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:08,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397828769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:08,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:08,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397828770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397828769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:08,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397828770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:08,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397828772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,853 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:08,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:08,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:08,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:08,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:08,856 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:08,856 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:08,856 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:08,856 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:08,857 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,857 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:08,858 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:08,858 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:08,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:08,858 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,858 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:08,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:08,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:08,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:08,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:08,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:08,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:08,859 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:08,859 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:08,859 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:08,860 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:08,860 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:08,860 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:08,860 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:08,862 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,862 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:08,863 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,863 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:08,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:08,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:08,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:08,864 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:08,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@767a7155[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:08,864 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,865 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,866 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:08,866 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:08,866 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:08,866 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,866 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:08,867 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:08,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:08,867 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:08,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:08,869 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:08,869 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:08,869 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:08,869 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:08,870 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:08,871 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,871 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:08,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,871 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:08,871 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:08,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,873 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:08,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397828874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:08,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397828874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:08,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397828874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:08,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397828874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,877 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,877 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:08,877 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:08,877 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:08,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:08,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:08,877 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:08,877 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:08,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:08,877 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:08,877 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:08,877 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:08,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:08,877 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397828875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,878 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,878 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:08,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:08,977 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:08,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:08,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:08,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:08,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:08,982 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:08,982 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:08,982 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:08,982 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:08,982 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,982 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:08,983 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:08,984 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:08,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:08,984 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,984 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:08,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:08,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:08,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:08,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:08,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:08,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:08,985 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:08,985 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:08,985 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:08,985 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:08,985 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:08,985 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:08,985 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:08,985 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:08,986 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:08,986 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:08,986 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:08,986 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,986 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:08,986 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:08,987 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,987 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:08,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:08,987 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,987 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:08,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:08,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:08,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d0894de[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:08,989 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:08,989 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,990 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:08,990 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:08,990 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:08,990 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:08,990 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,990 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:08,992 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:08,993 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,993 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:08,993 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:08,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:08,995 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:08,995 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:08,995 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:08,995 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:08,995 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:08,997 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,997 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:08,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:08,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,997 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:08,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:08,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:08,997 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:08,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:08,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:08,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:09,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:09,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:09,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:09,002 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:09,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:09,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:09,002 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:09,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:09,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:09,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:09,002 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:09,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397829076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397829077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397829077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397829077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397829078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,103 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:09,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:09,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:09,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:09,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:09,106 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:09,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:09,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:09,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:09,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,106 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,107 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:09,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:09,108 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,108 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,108 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:09,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:09,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:09,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:09,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:09,109 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:09,109 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:09,109 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:09,109 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:09,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:09,110 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:09,110 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:09,110 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:09,110 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:09,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:09,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:09,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:09,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:09,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,115 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:09,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,115 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:09,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:09,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:09,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c64995c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:09,117 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:09,117 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,118 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,118 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:09,118 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:09,118 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:09,118 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,118 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:09,119 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:09,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:09,119 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:09,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:09,122 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:09,122 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:09,122 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:09,122 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:09,122 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:09,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:09,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,124 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:09,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:09,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:09,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,128 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,128 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,128 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:09,128 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:09,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:09,128 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:09,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:09,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:09,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:09,128 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:09,128 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,128 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,128 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,129 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,129 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:09,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:09,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:09,129 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:09,193 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041326c134b00ff7495fbc9295c17d7cd7bf to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041326c134b00ff7495fbc9295c17d7cd7bf 2023-04-13 14:56:09,208 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/6c5ea36e7d5e483f90aaf42fa17eea93 2023-04-13 14:56:09,229 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:09,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:09,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:09,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:09,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:09,232 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:09,232 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:09,232 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:09,232 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:09,233 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,233 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,234 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:09,234 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,234 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,234 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:09,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:09,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:09,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:09,235 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:09,235 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:09,235 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:09,235 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:09,235 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:09,235 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:09,235 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:09,236 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:09,236 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:09,236 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:09,236 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:09,236 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,236 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:09,236 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:09,237 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,237 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:09,237 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,237 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:09,239 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:09,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6294a5dc[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:09,239 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,239 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:09,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:09,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:09,240 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,240 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:09,242 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:09,242 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:09,242 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:09,242 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:09,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:09,244 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:09,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:09,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:09,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:09,244 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:09,245 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,245 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:09,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,245 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:09,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:09,246 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,247 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,249 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,249 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,249 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,249 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,249 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,249 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:09,249 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:09,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:09,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:09,249 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:09,249 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:09,250 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,250 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:09,250 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:09,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:09,350 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:09,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:09,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:09,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:09,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:09,353 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:09,353 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:09,353 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:09,353 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:09,354 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,354 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,355 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:09,355 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:09,355 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,355 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:09,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:09,358 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:09,358 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:09,358 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:09,358 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:09,361 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,361 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:09,361 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,361 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47c19002[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:09,363 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:09,363 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,364 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,364 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:09,364 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:09,364 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:09,364 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,364 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:09,365 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:09,366 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:09,366 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:09,366 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:09,367 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:09,368 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:09,368 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:09,368 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:09,368 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:09,369 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,369 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,369 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:09,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:09,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:09,369 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:09,371 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,374 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:09,374 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:09,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:09,374 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,374 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,374 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:09,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:09,374 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:09,374 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,374 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,374 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,374 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,374 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:09,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:09,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:09,375 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:09,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397829379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397829380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397829380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397829381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397829382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,475 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:09,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:09,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:09,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:09,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:09,478 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:09,478 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:09,478 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:09,478 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:09,479 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,479 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,480 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:09,480 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:09,481 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,481 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:09,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:09,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:09,482 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:09,482 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:09,482 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:09,482 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:09,483 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:09,483 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,483 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:09,483 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:09,484 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:09,484 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,485 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:09,487 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:09,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e565240[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:09,487 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,491 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,491 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:09,491 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:09,491 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:09,491 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,492 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:09,493 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:09,493 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:09,493 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:09,493 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:09,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:09,496 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:09,496 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:09,496 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:09,496 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:09,496 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:09,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,497 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:09,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:09,498 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:09,499 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,501 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,501 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,501 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:09,501 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:09,501 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:09,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:09,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:09,501 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,502 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,502 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,502 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,502 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,502 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:09,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:09,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:09,502 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:09,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:09,602 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:09,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:09,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:09,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:09,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:09,606 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:09,606 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:09,606 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:09,606 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:09,606 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,606 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,608 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:09,608 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:09,608 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,608 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:09,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:09,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:09,609 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:09,609 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:09,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:09,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:09,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:09,610 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:09,610 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:09,610 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:09,610 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:09,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:09,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:09,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:09,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:09,612 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,612 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:09,612 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:09,614 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:09,615 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26a08aaf[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:09,616 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,616 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:09,616 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:09,616 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:09,616 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,616 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:09,618 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,618 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,618 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:09,618 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:09,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:09,621 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:09,621 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:09,621 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:09,621 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:09,621 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:09,623 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,623 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,623 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:09,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:09,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,624 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:09,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,626 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:09,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:09,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:09,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:09,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:09,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:09,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:09,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,628 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:09,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:09,628 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:09,628 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:09,629 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/a614ffd5241d4a54a5bc01da31cff56f 2023-04-13 14:56:09,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:09,638 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/58b9b3f7317342bc83b3d6d48ad17838 2023-04-13 14:56:09,641 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/6c5ea36e7d5e483f90aaf42fa17eea93 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/6c5ea36e7d5e483f90aaf42fa17eea93 2023-04-13 14:56:09,645 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/6c5ea36e7d5e483f90aaf42fa17eea93, entries=150, sequenceid=15, filesize=25.1 K 2023-04-13 14:56:09,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/a614ffd5241d4a54a5bc01da31cff56f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/a614ffd5241d4a54a5bc01da31cff56f 2023-04-13 14:56:09,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/a614ffd5241d4a54a5bc01da31cff56f, entries=150, sequenceid=15, filesize=11.6 K 2023-04-13 14:56:09,649 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/58b9b3f7317342bc83b3d6d48ad17838 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/58b9b3f7317342bc83b3d6d48ad17838 2023-04-13 14:56:09,652 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/58b9b3f7317342bc83b3d6d48ad17838, entries=150, sequenceid=15, filesize=11.6 K 2023-04-13 14:56:09,653 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 915ms, sequenceid=15, compaction requested=false 2023-04-13 14:56:09,653 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:09,728 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:09,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:09,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:09,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:09,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:09,733 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:09,733 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:09,733 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:09,733 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:09,733 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,734 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,735 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:09,735 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:09,735 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,735 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:09,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:09,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:09,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:09,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:09,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:09,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:09,737 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:09,737 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:09,737 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:09,737 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:09,737 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-13 14:56:09,737 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:09,737 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:09,737 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:09,738 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:09,738 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:09,738 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:09,753 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413ecd84892f00d4fe3a948d9587a9958ed to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ecd84892f00d4fe3a948d9587a9958ed 2023-04-13 14:56:09,758 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/4fa4795a9e3e4d5a9436db1b050dde28 2023-04-13 14:56:09,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:09,884 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:09,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397829888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397829888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397829889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397829889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397829890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397829991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397829992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397829993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397829993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:09,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:09,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397829993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,169 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/4055fca866454e39adc5d6de31a43884 2023-04-13 14:56:10,178 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/56294651dbe248ea8f12152dad922271 2023-04-13 14:56:10,182 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/4fa4795a9e3e4d5a9436db1b050dde28 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/4fa4795a9e3e4d5a9436db1b050dde28 2023-04-13 14:56:10,185 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/4fa4795a9e3e4d5a9436db1b050dde28, entries=150, sequenceid=52, filesize=25.1 K 2023-04-13 14:56:10,185 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/4055fca866454e39adc5d6de31a43884 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/4055fca866454e39adc5d6de31a43884 2023-04-13 14:56:10,189 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/4055fca866454e39adc5d6de31a43884, entries=150, sequenceid=52, filesize=11.6 K 2023-04-13 14:56:10,190 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/56294651dbe248ea8f12152dad922271 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/56294651dbe248ea8f12152dad922271 2023-04-13 14:56:10,193 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/56294651dbe248ea8f12152dad922271, entries=150, sequenceid=52, filesize=11.6 K 2023-04-13 14:56:10,194 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 457ms, sequenceid=52, compaction requested=false 2023-04-13 14:56:10,194 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:10,194 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:10,194 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:10,194 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:10,194 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,194 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:10,194 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:10,196 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,196 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:10,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 14:56:10,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:10,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:10,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:10,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:10,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:10,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:10,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:10,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:10,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:10,198 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,198 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:10,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:10,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@615d71e2[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:10,200 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:10,200 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,202 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,202 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,202 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:10,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:10,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:10,204 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:10,205 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:10,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:10,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:10,209 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:10,209 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:10,209 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:10,209 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:10,209 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:10,210 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,210 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,210 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:10,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:10,211 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,213 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,215 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413e89caafdc4b24ba3af0601638ef6391f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e89caafdc4b24ba3af0601638ef6391f 2023-04-13 14:56:10,215 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:10,215 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:10,215 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:10,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:10,215 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,215 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,216 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,216 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,216 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,216 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:10,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:10,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:10,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:10,216 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,216 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:10,216 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:10,221 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/2bea3ad4c2e3488797ca75b983c8414b 2023-04-13 14:56:10,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397830217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9558, totalHeapSize=21976, min timestamp=1681397770212, max timestamp=1681397770220Num uniques 859; 2023-04-13 14:56:10,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39328, new segment data size=9558, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:56:10,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681397770212, max timestamp=1681397770220Num uniques 859; 2023-04-13 14:56:10,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:56:10,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681397770212, max timestamp=1681397770220Num uniques 859; 2023-04-13 14:56:10,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:56:10,233 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/89004ef2a1684bca8be476237134f1f3 2023-04-13 14:56:10,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397830239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397830239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397830240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397830243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,316 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:10,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:10,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:10,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:10,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:10,320 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:10,320 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:10,320 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:10,320 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:10,320 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,320 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,322 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:10,322 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:10,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:10,323 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,323 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:10,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:10,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:10,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:10,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:10,324 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:10,324 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:10,325 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:10,325 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:10,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:10,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:10,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:10,325 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:10,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397830324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,326 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:10,327 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,327 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:10,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:10,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a57bc4b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:10,328 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:10,328 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,330 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,330 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,330 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:10,330 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:10,330 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:10,330 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:10,332 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:10,333 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:10,333 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:10,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:10,336 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:10,336 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:10,336 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:10,336 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:10,336 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:10,341 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,341 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,341 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:10,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:10,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:10,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:10,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:10,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,343 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,344 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,347 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:10,347 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:10,348 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:10,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:10,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:10,347 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:10,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:10,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:10,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397830346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,348 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:10,348 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:10,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397830346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,348 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,349 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,349 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:10,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:10,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397830346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397830348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,448 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:10,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:10,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:10,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:10,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:10,452 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:10,452 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:10,452 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:10,452 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:10,452 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,452 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,453 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,453 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:10,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:10,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:10,454 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,454 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:10,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:10,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:10,455 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:10,455 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:10,455 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:10,455 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:10,456 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:10,457 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,457 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:10,457 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,457 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:10,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:10,459 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:10,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@665f2f13[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:10,459 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,466 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,466 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:10,466 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,466 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:10,466 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:10,466 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:10,467 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,467 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:10,467 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:10,468 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:10,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:10,470 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:10,470 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:10,470 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:10,470 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:10,470 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:10,471 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,471 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,472 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:10,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,472 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:10,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,473 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:10,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,475 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:10,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:10,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:10,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:10,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:10,475 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:10,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:10,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:10,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:10,476 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,476 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,476 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:10,476 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,476 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:10,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397830528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397830549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397830550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397830551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397830551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,553 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 14:56:10,576 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:10,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:10,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:10,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:10,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:10,580 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:10,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:10,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:10,580 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:10,584 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,584 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,586 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,586 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:10,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:10,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:10,586 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,586 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:10,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:10,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:10,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:10,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:10,600 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:10,600 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:10,600 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:10,601 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:10,601 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:10,601 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:10,601 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:10,601 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:10,601 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:10,601 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:10,601 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:10,601 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,601 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:10,602 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:10,603 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,603 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:10,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:10,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:10,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44883485[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:10,606 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:10,606 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,607 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,607 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:10,607 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:10,607 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:10,608 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,608 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:10,609 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:10,609 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:10,610 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:10,609 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:10,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:10,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:10,613 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:10,613 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:10,613 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:10,613 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:10,613 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:10,615 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,615 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:10,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,616 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:10,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:10,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:10,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,620 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:10,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,622 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,625 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,625 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:10,625 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:10,625 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,625 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,625 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,625 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,625 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:10,625 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,625 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:10,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:10,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:10,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:10,625 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:10,625 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:10,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:10,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/0ace186f878e4475a3030f598e800c2f 2023-04-13 14:56:10,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/2bea3ad4c2e3488797ca75b983c8414b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2bea3ad4c2e3488797ca75b983c8414b 2023-04-13 14:56:10,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2bea3ad4c2e3488797ca75b983c8414b, entries=150, sequenceid=63, filesize=25.1 K 2023-04-13 14:56:10,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/89004ef2a1684bca8be476237134f1f3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/89004ef2a1684bca8be476237134f1f3 2023-04-13 14:56:10,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/89004ef2a1684bca8be476237134f1f3, entries=150, sequenceid=63, filesize=11.6 K 2023-04-13 14:56:10,655 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/0ace186f878e4475a3030f598e800c2f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/0ace186f878e4475a3030f598e800c2f 2023-04-13 14:56:10,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/0ace186f878e4475a3030f598e800c2f, entries=150, sequenceid=63, filesize=11.6 K 2023-04-13 14:56:10,659 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 463ms, sequenceid=63, compaction requested=true 2023-04-13 14:56:10,660 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:10,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:10,660 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:56:10,660 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:56:10,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:10,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:10,662 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:56:10,662 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/A is initiating minor compaction (all files) 2023-04-13 14:56:10,662 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/A in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:10,663 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/6c5ea36e7d5e483f90aaf42fa17eea93, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/4fa4795a9e3e4d5a9436db1b050dde28, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2bea3ad4c2e3488797ca75b983c8414b] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=75.4 K 2023-04-13 14:56:10,663 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:56:10,663 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/B is initiating minor compaction (all files) 2023-04-13 14:56:10,663 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/B in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:10,663 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 6c5ea36e7d5e483f90aaf42fa17eea93, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681397768732 2023-04-13 14:56:10,663 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/a614ffd5241d4a54a5bc01da31cff56f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/4055fca866454e39adc5d6de31a43884, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/89004ef2a1684bca8be476237134f1f3] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=34.8 K 2023-04-13 14:56:10,664 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 4fa4795a9e3e4d5a9436db1b050dde28, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681397768762 2023-04-13 14:56:10,664 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting a614ffd5241d4a54a5bc01da31cff56f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681397768732 2023-04-13 14:56:10,665 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 2bea3ad4c2e3488797ca75b983c8414b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681397769888 2023-04-13 14:56:10,665 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 4055fca866454e39adc5d6de31a43884, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681397768762 2023-04-13 14:56:10,665 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 89004ef2a1684bca8be476237134f1f3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681397769888 2023-04-13 14:56:10,677 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#B#compaction#524 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:10,701 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#A#compaction#523 average throughput is 0.68 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:10,726 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:10,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:10,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:10,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:10,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:10,729 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:10,729 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:10,729 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:10,729 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:10,730 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,730 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,731 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,731 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:10,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:10,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:10,731 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,731 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:10,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:10,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:10,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:10,732 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:10,732 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:10,732 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:10,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:10,736 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:10,736 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:10,736 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:10,736 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-13 14:56:10,736 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:10,736 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:10,736 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:10,736 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:10,737 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:10,737 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:10,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:10,836 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:10,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397830855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397830855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397830855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397830857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397830858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397830959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397830962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:10,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:10,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397830964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:11,114 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/d7316b8d15924fe3906b684c1690cf3a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/d7316b8d15924fe3906b684c1690cf3a 2023-04-13 14:56:11,117 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/B of 7f6fba518b666b8072f5bc2fa5e4a5d5 into d7316b8d15924fe3906b684c1690cf3a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:11,118 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:11,118 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/B, priority=13, startTime=1681397770660; duration=0sec 2023-04-13 14:56:11,118 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:11,118 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:56:11,119 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:56:11,119 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/C is initiating minor compaction (all files) 2023-04-13 14:56:11,119 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/C in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:11,119 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/58b9b3f7317342bc83b3d6d48ad17838, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/56294651dbe248ea8f12152dad922271, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/0ace186f878e4475a3030f598e800c2f] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=34.8 K 2023-04-13 14:56:11,119 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 58b9b3f7317342bc83b3d6d48ad17838, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681397768732 2023-04-13 14:56:11,120 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 56294651dbe248ea8f12152dad922271, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681397768762 2023-04-13 14:56:11,120 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 0ace186f878e4475a3030f598e800c2f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681397769888 2023-04-13 14:56:11,124 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#C#compaction#526 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:11,130 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/ef23692be430484a8a3f9bfd51a9c33b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ef23692be430484a8a3f9bfd51a9c33b 2023-04-13 14:56:11,133 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/e56867305bc945ff9130f54bae9d8045 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/e56867305bc945ff9130f54bae9d8045 2023-04-13 14:56:11,134 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/A of 7f6fba518b666b8072f5bc2fa5e4a5d5 into ef23692be430484a8a3f9bfd51a9c33b(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:11,134 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:11,134 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/A, priority=13, startTime=1681397770660; duration=0sec 2023-04-13 14:56:11,135 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:11,137 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/C of 7f6fba518b666b8072f5bc2fa5e4a5d5 into e56867305bc945ff9130f54bae9d8045(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:11,137 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:11,137 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/C, priority=13, startTime=1681397770660; duration=0sec 2023-04-13 14:56:11,137 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:11,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:11,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397831162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:11,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:11,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397831165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:11,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:11,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397831168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:11,171 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413c80acfebfc304109a00e31145fe9f79c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c80acfebfc304109a00e31145fe9f79c 2023-04-13 14:56:11,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:11,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397831359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:11,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:11,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397831363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:11,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:11,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397831467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:11,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:11,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397831469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:11,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:11,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397831471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:11,578 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/3eaa196ef2524bf9be3d18858c33a8a6 2023-04-13 14:56:11,588 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/618fe22ba3df49089c13064144185bf1 2023-04-13 14:56:11,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:11,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397831973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:11,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:11,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397831975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:11,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:11,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397831976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:11,998 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/22951d6e11ac4b38851e3cb5feedf135 2023-04-13 14:56:12,002 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/3eaa196ef2524bf9be3d18858c33a8a6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/3eaa196ef2524bf9be3d18858c33a8a6 2023-04-13 14:56:12,005 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/3eaa196ef2524bf9be3d18858c33a8a6, entries=150, sequenceid=98, filesize=25.1 K 2023-04-13 14:56:12,006 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/618fe22ba3df49089c13064144185bf1 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/618fe22ba3df49089c13064144185bf1 2023-04-13 14:56:12,010 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/618fe22ba3df49089c13064144185bf1, entries=150, sequenceid=98, filesize=11.6 K 2023-04-13 14:56:12,010 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/22951d6e11ac4b38851e3cb5feedf135 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/22951d6e11ac4b38851e3cb5feedf135 2023-04-13 14:56:12,017 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/22951d6e11ac4b38851e3cb5feedf135, entries=150, sequenceid=98, filesize=11.6 K 2023-04-13 14:56:12,018 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 1282ms, sequenceid=98, compaction requested=false 2023-04-13 14:56:12,018 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:12,018 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:12,018 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:12,020 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,020 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:12,020 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:12,020 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:12,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:12,022 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:12,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ed09f97[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:12,022 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:12,023 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:12,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:12,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:12,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:12,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:12,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:12,023 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,023 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:12,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:12,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:12,024 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:12,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:12,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:12,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:12,027 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:12,027 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:12,027 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:12,027 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:12,027 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:12,029 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:12,029 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:12,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:12,029 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:12,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:12,030 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:12,032 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,036 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,036 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:12,036 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:12,036 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:12,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:12,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:12,036 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:12,037 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,037 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:12,037 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:12,037 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:12,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:12,037 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:12,036 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:12,037 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:12,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:12,137 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:12,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:12,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:12,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:12,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:12,140 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:12,140 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:12,140 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:12,140 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:12,141 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:12,141 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:12,142 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:12,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:12,142 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:12,142 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,143 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:12,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:12,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:12,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:12,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:12,144 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:12,144 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:12,144 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 14:56:12,144 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:12,144 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:12,144 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:12,144 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:12,144 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:12,144 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:12,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:12,368 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:12,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397772378, max timestamp=1681397772392Num uniques 859; 2023-04-13 14:56:12,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397772378, max timestamp=1681397772392Num uniques 859; 2023-04-13 14:56:12,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:56:12,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397772378, max timestamp=1681397772392Num uniques 859; 2023-04-13 14:56:12,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:12,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:56:12,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397832392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:56:12,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:12,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397832451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:12,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397832496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:12,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397832556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,559 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413279e2106c3d141e18019cfd483f3cd70 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413279e2106c3d141e18019cfd483f3cd70 2023-04-13 14:56:12,565 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=112, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/dd1839b084354bf3a3984bc534d1feaa 2023-04-13 14:56:12,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:12,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397832701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:12,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397832759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,977 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/e5626b02956c4d79bd065c28768651d2 2023-04-13 14:56:12,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:12,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397832978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:12,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397832982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:12,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:12,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397832984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:13,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397833006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:13,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397833061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,387 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/373df13911d340feb89204ff4ebeb3d9 2023-04-13 14:56:13,391 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/dd1839b084354bf3a3984bc534d1feaa as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/dd1839b084354bf3a3984bc534d1feaa 2023-04-13 14:56:13,394 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/dd1839b084354bf3a3984bc534d1feaa, entries=150, sequenceid=112, filesize=25.1 K 2023-04-13 14:56:13,395 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/e5626b02956c4d79bd065c28768651d2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/e5626b02956c4d79bd065c28768651d2 2023-04-13 14:56:13,398 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/e5626b02956c4d79bd065c28768651d2, entries=150, sequenceid=112, filesize=11.6 K 2023-04-13 14:56:13,399 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/373df13911d340feb89204ff4ebeb3d9 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/373df13911d340feb89204ff4ebeb3d9 2023-04-13 14:56:13,403 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/373df13911d340feb89204ff4ebeb3d9, entries=150, sequenceid=112, filesize=11.6 K 2023-04-13 14:56:13,404 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 1260ms, sequenceid=112, compaction requested=true 2023-04-13 14:56:13,404 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:13,404 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:13,404 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:13,404 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:13,404 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,404 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:13,404 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:13,407 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,407 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:13,408 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,408 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:13,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:13,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f2c7e2[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:13,410 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:13,410 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,411 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,411 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:13,411 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:13,411 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,411 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:13,411 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:13,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:13,413 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:13,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:13,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:13,415 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:13,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:13,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:13,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:13,415 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:13,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:13,417 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:13,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:13,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:13,418 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,420 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,423 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,423 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,423 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:13,423 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:13,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:13,423 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:13,423 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:13,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:13,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:13,423 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,424 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,424 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,424 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,424 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:13,424 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:13,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:13,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:13,512 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-13 14:56:13,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:13,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:13,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:13,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:13,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:13,513 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:13,524 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:13,524 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413cdef4e3b6db64635b07e07c019f45ce6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413cdef4e3b6db64635b07e07c019f45ce6 2023-04-13 14:56:13,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:13,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:13,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:13,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:13,534 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:13,534 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:13,534 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:13,534 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:13,534 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,534 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,536 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:13,536 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,536 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=148, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/3e166f2d889f454b81e72c0af27fd2b4 2023-04-13 14:56:13,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:13,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:13,536 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:13,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:13,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:13,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:13,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:13,538 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:13,538 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:13,539 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:13,539 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:13,539 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:13,539 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:13,539 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,539 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:13,539 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:13,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,540 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,541 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:13,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:13,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46ca4c78[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:13,543 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:13,543 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,544 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,544 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:13,544 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:13,544 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:13,544 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,544 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:13,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:13,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397833541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,546 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:13,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:13,548 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:13,548 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:13,548 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:13,548 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:13,548 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:13,550 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,550 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:13,550 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:13,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:13,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,551 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,552 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:13,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:13,555 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:13,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:13,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:13,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:13,555 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,555 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:13,555 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:13,555 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:13,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:13,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397833564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:13,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397833646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,655 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:13,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:13,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:13,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:13,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:13,659 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:13,659 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:13,659 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:13,659 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:13,659 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,659 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,660 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,660 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:13,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:13,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:13,661 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,661 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:13,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:13,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:13,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:13,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:13,662 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:13,662 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:13,662 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:13,662 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:13,662 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:13,662 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:13,662 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:13,662 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:13,662 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:13,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:13,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:13,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:13,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:13,665 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,665 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:13,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:13,665 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,665 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:13,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:13,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:13,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:13,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:13,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43636438[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:13,667 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:13,667 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,668 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,668 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,668 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:13,668 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:13,668 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:13,668 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:13,669 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,669 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:13,669 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:13,669 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:13,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:13,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:13,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:13,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:13,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:13,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:13,671 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:13,672 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:13,672 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:13,672 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:13,672 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:13,673 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,673 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,673 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:13,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:13,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:13,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:13,674 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:13,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,675 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:13,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:13,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:13,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:13,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:13,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:13,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:13,680 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:13,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:13,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:13,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:13,681 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:13,681 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:13,781 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:13,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:13,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:13,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:13,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:13,785 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:13,785 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:13,785 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:13,785 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:13,785 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,785 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,787 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:13,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:13,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:13,787 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,787 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,787 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:13,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:13,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:13,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:13,788 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:13,789 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:13,789 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:13,789 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:13,789 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:13,789 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:13,789 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:13,789 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:13,789 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:13,789 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:13,789 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:13,789 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,789 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:13,789 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:13,791 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,791 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:13,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:13,792 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,792 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:13,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:13,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:13,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:13,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:13,794 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:13,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@98567e4[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:13,794 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,795 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,795 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:13,795 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:13,795 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:13,795 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,795 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:13,798 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:13,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:13,798 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:13,798 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:13,798 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:13,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:13,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:13,801 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:13,801 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:13,801 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:13,801 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:13,801 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:13,803 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,803 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,803 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:13,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:13,804 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:13,805 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,807 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,807 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,807 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,807 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:13,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:13,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:13,807 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:13,807 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,807 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,807 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,807 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:13,807 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:13,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:13,807 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:13,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:13,807 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:13,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:13,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397833849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,908 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:13,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:13,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:13,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:13,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:13,911 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:13,911 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:13,911 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:13,911 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:13,912 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,912 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,914 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:13,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:13,914 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,914 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,914 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:13,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:13,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:13,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:13,915 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:13,915 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:13,915 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:13,915 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:13,915 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:13,916 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:13,916 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:13,916 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:13,916 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:13,916 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:13,916 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:13,916 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,916 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:13,916 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:13,919 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:13,919 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:13,919 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,919 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:13,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:13,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46076ce1[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:13,921 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:13,921 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,922 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,922 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:13,922 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,922 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:13,922 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:13,922 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:13,923 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,923 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:13,923 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:13,923 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:13,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:13,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:13,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:13,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:13,926 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:13,926 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:13,926 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:13,926 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:13,926 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:13,928 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,928 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:13,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,928 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:13,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:13,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:13,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:13,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:13,928 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:13,930 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:13,932 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,932 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:13,932 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:13,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:13,933 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:13,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:13,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:13,933 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:13,933 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:13,933 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,933 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:13,933 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:13,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:13,933 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:13,933 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:13,933 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:13,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:13,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:13,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/653ff3a529f14970949c9cb2fc713c89 2023-04-13 14:56:13,967 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/ad7e59d52ab046ea9c5e7e2f4841e1a3 2023-04-13 14:56:13,971 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/3e166f2d889f454b81e72c0af27fd2b4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/3e166f2d889f454b81e72c0af27fd2b4 2023-04-13 14:56:13,974 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/3e166f2d889f454b81e72c0af27fd2b4, entries=200, sequenceid=148, filesize=32.1 K 2023-04-13 14:56:13,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/653ff3a529f14970949c9cb2fc713c89 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/653ff3a529f14970949c9cb2fc713c89 2023-04-13 14:56:13,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/653ff3a529f14970949c9cb2fc713c89, entries=150, sequenceid=148, filesize=11.8 K 2023-04-13 14:56:13,979 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/ad7e59d52ab046ea9c5e7e2f4841e1a3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/ad7e59d52ab046ea9c5e7e2f4841e1a3 2023-04-13 14:56:13,983 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/ad7e59d52ab046ea9c5e7e2f4841e1a3, entries=150, sequenceid=148, filesize=11.8 K 2023-04-13 14:56:13,984 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 472ms, sequenceid=148, compaction requested=true 2023-04-13 14:56:13,984 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:13,984 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:13,984 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:56:13,984 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:56:13,984 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:13,984 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:13,985 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110227 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:56:13,985 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:56:13,985 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/A is initiating minor compaction (all files) 2023-04-13 14:56:13,985 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/B is initiating minor compaction (all files) 2023-04-13 14:56:13,985 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/A in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:13,985 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/B in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:13,985 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ef23692be430484a8a3f9bfd51a9c33b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/3eaa196ef2524bf9be3d18858c33a8a6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/dd1839b084354bf3a3984bc534d1feaa, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/3e166f2d889f454b81e72c0af27fd2b4] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=107.6 K 2023-04-13 14:56:13,985 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/d7316b8d15924fe3906b684c1690cf3a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/618fe22ba3df49089c13064144185bf1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/e5626b02956c4d79bd065c28768651d2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/653ff3a529f14970949c9cb2fc713c89] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=46.7 K 2023-04-13 14:56:13,986 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting ef23692be430484a8a3f9bfd51a9c33b, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681397769888 2023-04-13 14:56:13,986 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting d7316b8d15924fe3906b684c1690cf3a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681397769888 2023-04-13 14:56:13,986 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 3eaa196ef2524bf9be3d18858c33a8a6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681397770238 2023-04-13 14:56:13,986 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 618fe22ba3df49089c13064144185bf1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681397770238 2023-04-13 14:56:13,986 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting dd1839b084354bf3a3984bc534d1feaa, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1681397770853 2023-04-13 14:56:13,987 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting e5626b02956c4d79bd065c28768651d2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1681397770853 2023-04-13 14:56:13,987 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 3e166f2d889f454b81e72c0af27fd2b4, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681397772407 2023-04-13 14:56:13,987 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 653ff3a529f14970949c9cb2fc713c89, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681397772432 2023-04-13 14:56:14,013 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#B#compaction#536 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:14,033 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:14,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:14,035 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#A#compaction#535 average throughput is 0.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:14,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:14,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:14,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:14,049 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:14,049 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:14,049 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:14,049 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:14,050 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:14,050 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:14,051 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:14,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:14,052 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:14,052 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:14,052 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:14,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:14,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:14,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:14,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:14,053 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:14,053 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:14,053 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:14,056 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:14,056 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:14,056 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:14,056 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:14,056 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:56:14,056 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:14,056 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:14,056 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:14,056 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:14,057 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:14,057 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:14,096 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413b986939c336b487b89cc1165995a29a8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b986939c336b487b89cc1165995a29a8 2023-04-13 14:56:14,110 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=158, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/70b80995c6414e138af645b0b61d3e3c 2023-04-13 14:56:14,112 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/d3cd624fa126414bb61f5ce3cba6779b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/d3cd624fa126414bb61f5ce3cba6779b 2023-04-13 14:56:14,116 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/A of 7f6fba518b666b8072f5bc2fa5e4a5d5 into d3cd624fa126414bb61f5ce3cba6779b(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:14,116 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:14,117 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/A, priority=12, startTime=1681397773984; duration=0sec 2023-04-13 14:56:14,117 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:14,117 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:56:14,118 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:56:14,118 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/C is initiating minor compaction (all files) 2023-04-13 14:56:14,118 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/C in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:14,119 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/e56867305bc945ff9130f54bae9d8045, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/22951d6e11ac4b38851e3cb5feedf135, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/373df13911d340feb89204ff4ebeb3d9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/ad7e59d52ab046ea9c5e7e2f4841e1a3] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=46.7 K 2023-04-13 14:56:14,119 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting e56867305bc945ff9130f54bae9d8045, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681397769888 2023-04-13 14:56:14,119 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 22951d6e11ac4b38851e3cb5feedf135, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681397770238 2023-04-13 14:56:14,120 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 373df13911d340feb89204ff4ebeb3d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1681397770853 2023-04-13 14:56:14,121 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting ad7e59d52ab046ea9c5e7e2f4841e1a3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681397772432 2023-04-13 14:56:14,136 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#C#compaction#539 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:14,138 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/c1446c8073a44d10a7bd56d98e984abc 2023-04-13 14:56:14,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:14,163 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:14,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397774203, max timestamp=1681397774226Num uniques 859; 2023-04-13 14:56:14,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397774203, max timestamp=1681397774226Num uniques 859; 2023-04-13 14:56:14,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:14,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:14,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397774203, max timestamp=1681397774226Num uniques 859; 2023-04-13 14:56:14,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:14,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:14,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397834290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:14,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397834395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,446 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/da64db90e06147edadd5dd1872f7662d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/da64db90e06147edadd5dd1872f7662d 2023-04-13 14:56:14,451 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/B of 7f6fba518b666b8072f5bc2fa5e4a5d5 into da64db90e06147edadd5dd1872f7662d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:14,451 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:14,451 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/B, priority=12, startTime=1681397773984; duration=0sec 2023-04-13 14:56:14,451 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:14,554 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/804ee7008aef4bc1a272c0baefa6d56d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/804ee7008aef4bc1a272c0baefa6d56d 2023-04-13 14:56:14,558 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/C of 7f6fba518b666b8072f5bc2fa5e4a5d5 into 804ee7008aef4bc1a272c0baefa6d56d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:14,558 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:14,558 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/C, priority=12, startTime=1681397773984; duration=0sec 2023-04-13 14:56:14,558 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:14,570 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/8c96c4d0b421444dae0c3f2a454614f5 2023-04-13 14:56:14,573 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/70b80995c6414e138af645b0b61d3e3c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70b80995c6414e138af645b0b61d3e3c 2023-04-13 14:56:14,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:14,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397834574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,576 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70b80995c6414e138af645b0b61d3e3c, entries=150, sequenceid=158, filesize=25.3 K 2023-04-13 14:56:14,577 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/c1446c8073a44d10a7bd56d98e984abc as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c1446c8073a44d10a7bd56d98e984abc 2023-04-13 14:56:14,580 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c1446c8073a44d10a7bd56d98e984abc, entries=150, sequenceid=158, filesize=11.8 K 2023-04-13 14:56:14,581 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/8c96c4d0b421444dae0c3f2a454614f5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/8c96c4d0b421444dae0c3f2a454614f5 2023-04-13 14:56:14,584 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/8c96c4d0b421444dae0c3f2a454614f5, entries=150, sequenceid=158, filesize=11.8 K 2023-04-13 14:56:14,585 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 529ms, sequenceid=158, compaction requested=false 2023-04-13 14:56:14,585 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:14,585 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:14,585 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:14,585 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:14,585 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,585 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:14,585 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:14,587 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:14,587 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:14,587 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:14,587 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:14,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:14,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@523a51d5[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:14,590 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:14,590 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:14,591 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:14,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:14,591 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:14,591 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:14,591 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:14,591 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,591 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:14,593 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:14,593 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:14,593 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:14,593 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:14,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:14,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:14,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:14,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:14,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:14,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:14,596 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:14,596 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:14,596 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:14,596 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:14,596 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:14,597 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:14,597 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:14,597 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:14,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:14,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:14,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:14,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:14,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:14,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:14,598 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:14,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:14,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:14,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:14,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:14,600 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:14,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:14,601 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-13 14:56:14,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:14,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:56:14,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:14,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:56:14,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:14,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:56:14,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:14,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,605 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:14,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:14,605 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:14,605 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:14,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:14,605 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:14,605 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:14,605 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:14,605 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,605 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:14,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:14,605 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:14,605 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:14,605 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:14,605 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:14,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:14,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:14,617 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304130756eba6cd1f42f29f570ec36b445ddb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304130756eba6cd1f42f29f570ec36b445ddb 2023-04-13 14:56:14,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:14,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38448 deadline: 1681397834628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,632 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=199, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/5b5b62429cb84914bbb15f9c929c8fe5 2023-04-13 14:56:14,654 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/aff4e27b80f7423383a2c4478a8736ab 2023-04-13 14:56:14,669 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/913fd4d71e4f43a48481a5c23328dd3e 2023-04-13 14:56:14,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/5b5b62429cb84914bbb15f9c929c8fe5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5b5b62429cb84914bbb15f9c929c8fe5 2023-04-13 14:56:14,677 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5b5b62429cb84914bbb15f9c929c8fe5, entries=200, sequenceid=199, filesize=32.1 K 2023-04-13 14:56:14,678 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/aff4e27b80f7423383a2c4478a8736ab as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/aff4e27b80f7423383a2c4478a8736ab 2023-04-13 14:56:14,682 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/aff4e27b80f7423383a2c4478a8736ab, entries=150, sequenceid=199, filesize=11.8 K 2023-04-13 14:56:14,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/913fd4d71e4f43a48481a5c23328dd3e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/913fd4d71e4f43a48481a5c23328dd3e 2023-04-13 14:56:14,686 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/913fd4d71e4f43a48481a5c23328dd3e, entries=150, sequenceid=199, filesize=11.8 K 2023-04-13 14:56:14,687 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 86ms, sequenceid=199, compaction requested=true 2023-04-13 14:56:14,687 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:14,687 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:14,687 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:14,687 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:56:14,687 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:56:14,687 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:14,691 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84918 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:56:14,691 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:56:14,691 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/A is initiating minor compaction (all files) 2023-04-13 14:56:14,691 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/B is initiating minor compaction (all files) 2023-04-13 14:56:14,691 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/A in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:14,691 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/B in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:14,691 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/d3cd624fa126414bb61f5ce3cba6779b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70b80995c6414e138af645b0b61d3e3c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5b5b62429cb84914bbb15f9c929c8fe5] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=82.9 K 2023-04-13 14:56:14,691 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/da64db90e06147edadd5dd1872f7662d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c1446c8073a44d10a7bd56d98e984abc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/aff4e27b80f7423383a2c4478a8736ab] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=35.5 K 2023-04-13 14:56:14,691 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting d3cd624fa126414bb61f5ce3cba6779b, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681397772432 2023-04-13 14:56:14,692 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting da64db90e06147edadd5dd1872f7662d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681397772432 2023-04-13 14:56:14,692 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 70b80995c6414e138af645b0b61d3e3c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681397773529 2023-04-13 14:56:14,692 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting c1446c8073a44d10a7bd56d98e984abc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681397773529 2023-04-13 14:56:14,693 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 5b5b62429cb84914bbb15f9c929c8fe5, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681397774275 2023-04-13 14:56:14,693 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting aff4e27b80f7423383a2c4478a8736ab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681397774284 2023-04-13 14:56:14,701 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#B#compaction#544 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:14,705 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:14,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:14,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:14,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:14,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:14,721 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:14,721 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:14,721 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:14,721 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:14,722 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:14,722 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:14,723 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#A#compaction#545 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:14,723 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,723 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:14,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:14,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:14,724 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:14,724 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:14,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:14,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:14,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:14,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:14,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:14,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:14,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:14,726 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:14,726 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:14,726 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:14,728 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:14,728 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:14,728 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:14,728 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:14,729 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 14:56:14,729 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:14,729 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:14,729 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:14,729 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:14,729 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:14,729 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:14,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:14,735 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:14,752 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/207be9deabcc41ca8afb1ac5066be5e2 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/207be9deabcc41ca8afb1ac5066be5e2 2023-04-13 14:56:14,757 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/A of 7f6fba518b666b8072f5bc2fa5e4a5d5 into 207be9deabcc41ca8afb1ac5066be5e2(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:14,758 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:14,758 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/A, priority=13, startTime=1681397774687; duration=0sec 2023-04-13 14:56:14,758 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:14,758 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:56:14,759 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:56:14,759 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/C is initiating minor compaction (all files) 2023-04-13 14:56:14,760 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/C in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:14,760 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/804ee7008aef4bc1a272c0baefa6d56d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/8c96c4d0b421444dae0c3f2a454614f5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/913fd4d71e4f43a48481a5c23328dd3e] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=35.5 K 2023-04-13 14:56:14,760 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 804ee7008aef4bc1a272c0baefa6d56d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681397772432 2023-04-13 14:56:14,760 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 8c96c4d0b421444dae0c3f2a454614f5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681397773529 2023-04-13 14:56:14,761 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 913fd4d71e4f43a48481a5c23328dd3e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681397774284 2023-04-13 14:56:14,769 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#C#compaction#547 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:14,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397774795, max timestamp=1681397774804Num uniques 859; 2023-04-13 14:56:14,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:14,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397774795, max timestamp=1681397774804Num uniques 859; 2023-04-13 14:56:14,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:14,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397774795, max timestamp=1681397774804Num uniques 859; 2023-04-13 14:56:14,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:14,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397774824, max timestamp=1681397774856Num uniques 1018; 2023-04-13 14:56:14,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397774824, max timestamp=1681397774856Num uniques 1018; 2023-04-13 14:56:14,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681397774824, max timestamp=1681397774856Num uniques 1018; 2023-04-13 14:56:14,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 14:56:14,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 14:56:14,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 14:56:14,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681397774902, max timestamp=1681397774916Num uniques 1027; 2023-04-13 14:56:14,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 14:56:14,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681397774902, max timestamp=1681397774916Num uniques 1027; 2023-04-13 14:56:14,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 14:56:14,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681397774902, max timestamp=1681397774916Num uniques 1027; 2023-04-13 14:56:14,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 14:56:14,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681397774959, max timestamp=1681397774973Num uniques 1036; 2023-04-13 14:56:14,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-13 14:56:14,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681397774959, max timestamp=1681397774973Num uniques 1036; 2023-04-13 14:56:14,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-13 14:56:14,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681397774959, max timestamp=1681397774973Num uniques 1036; 2023-04-13 14:56:14,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-13 14:56:15,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397835006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397835006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1681397774997, max timestamp=1681397775008Num uniques 1045; 2023-04-13 14:56:15,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=8920, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-13 14:56:15,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1681397774997, max timestamp=1681397775008Num uniques 1045; 2023-04-13 14:56:15,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=8920, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-13 14:56:15,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1681397774997, max timestamp=1681397775008Num uniques 1045; 2023-04-13 14:56:15,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=8920, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-13 14:56:15,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397835036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397835037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397835112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397835112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,116 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/1660bc04e00b4111b25cb2e2317cc7a5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/1660bc04e00b4111b25cb2e2317cc7a5 2023-04-13 14:56:15,121 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/B of 7f6fba518b666b8072f5bc2fa5e4a5d5 into 1660bc04e00b4111b25cb2e2317cc7a5(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:15,121 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:15,121 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/B, priority=13, startTime=1681397774687; duration=0sec 2023-04-13 14:56:15,121 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:15,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397835140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397835140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,151 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304135aed7df4d01b46b5aa0f88a21ad19fb2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304135aed7df4d01b46b5aa0f88a21ad19fb2 2023-04-13 14:56:15,157 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=208, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/ce4e3a9b6bb14438b241c203712fb40c 2023-04-13 14:56:15,176 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/6e0ffb6e98504efab5f28f2cb5ef7248 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6e0ffb6e98504efab5f28f2cb5ef7248 2023-04-13 14:56:15,180 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/C of 7f6fba518b666b8072f5bc2fa5e4a5d5 into 6e0ffb6e98504efab5f28f2cb5ef7248(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:15,180 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:15,180 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/C, priority=13, startTime=1681397774687; duration=0sec 2023-04-13 14:56:15,180 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:15,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397835317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397835318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397835343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397835345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,567 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/4a0c45f66df544099054ce033e05cb9d 2023-04-13 14:56:15,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397835620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397835621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397835647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:15,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397835649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,976 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/90d612a26d5e455dba9b350014d5ad33 2023-04-13 14:56:15,980 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/ce4e3a9b6bb14438b241c203712fb40c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ce4e3a9b6bb14438b241c203712fb40c 2023-04-13 14:56:15,984 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ce4e3a9b6bb14438b241c203712fb40c, entries=100, sequenceid=208, filesize=18.5 K 2023-04-13 14:56:15,984 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/4a0c45f66df544099054ce033e05cb9d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/4a0c45f66df544099054ce033e05cb9d 2023-04-13 14:56:15,988 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/4a0c45f66df544099054ce033e05cb9d, entries=100, sequenceid=208, filesize=9.4 K 2023-04-13 14:56:15,988 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/90d612a26d5e455dba9b350014d5ad33 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/90d612a26d5e455dba9b350014d5ad33 2023-04-13 14:56:15,992 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/90d612a26d5e455dba9b350014d5ad33, entries=100, sequenceid=208, filesize=9.4 K 2023-04-13 14:56:15,992 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 1264ms, sequenceid=208, compaction requested=false 2023-04-13 14:56:15,993 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:15,993 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:15,993 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:15,993 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:15,993 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,993 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:15,993 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:15,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:15,998 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:15,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:15,998 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:15,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:15,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:15,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:16,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:16,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a20bd0e[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:16,000 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:16,000 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:16,001 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:16,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:16,001 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,001 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:16,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:16,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:16,003 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:16,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:16,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:16,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:16,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:16,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:16,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:16,006 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:16,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:16,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:16,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:16,006 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:16,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:16,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:16,011 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:16,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:16,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:16,011 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:16,013 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:16,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,016 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:16,016 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,016 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:16,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:16,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:16,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:16,016 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:16,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:16,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:16,016 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:16,016 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:16,016 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:16,016 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,016 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:16,017 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:16,017 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:16,017 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:16,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:16,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:16,117 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:16,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:16,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:16,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:16,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:16,120 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:16,120 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:16,120 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:16,120 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:16,121 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:16,121 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:16,122 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:16,122 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:16,123 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,123 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:16,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:16,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:16,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:16,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:16,124 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:16,124 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:16,124 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:16,124 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:16,124 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:16,124 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:16,124 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:16,124 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=140.89 KB heapSize=401.53 KB 2023-04-13 14:56:16,125 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:16,125 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:16,125 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:16,125 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:16,125 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:16,125 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:16,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:16,127 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:16,140 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304135d572185edf74f04be98f72ea498c1ab to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304135d572185edf74f04be98f72ea498c1ab 2023-04-13 14:56:16,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:16,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397836141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:16,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397836141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:16,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397836153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:16,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397836157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:16,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397836245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:16,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397836246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:16,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397836448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:16,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397836449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,546 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=317, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/7f86114abf57477ca67b780a3dda4880 2023-04-13 14:56:16,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:16,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397836584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,588 DEBUG [Thread-3819] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:56:16,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:16,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397836752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:16,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397836754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,957 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/c8ae2684a539459ea0c94ec7118a602c 2023-04-13 14:56:16,966 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/5bdacb8d7a3d44d6ad0133bcd21c448e 2023-04-13 14:56:16,981 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/7f86114abf57477ca67b780a3dda4880 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/7f86114abf57477ca67b780a3dda4880 2023-04-13 14:56:16,986 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/7f86114abf57477ca67b780a3dda4880, entries=150, sequenceid=317, filesize=25.4 K 2023-04-13 14:56:16,987 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/c8ae2684a539459ea0c94ec7118a602c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c8ae2684a539459ea0c94ec7118a602c 2023-04-13 14:56:16,991 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c8ae2684a539459ea0c94ec7118a602c, entries=150, sequenceid=317, filesize=11.9 K 2023-04-13 14:56:16,991 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/5bdacb8d7a3d44d6ad0133bcd21c448e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/5bdacb8d7a3d44d6ad0133bcd21c448e 2023-04-13 14:56:16,994 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/5bdacb8d7a3d44d6ad0133bcd21c448e, entries=150, sequenceid=317, filesize=11.9 K 2023-04-13 14:56:16,995 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~401.48 KB/411120, currentSize=46.96 KB/48090 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 871ms, sequenceid=317, compaction requested=true 2023-04-13 14:56:16,995 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:16,995 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:16,995 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:16,995 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:16,995 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,995 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:16,995 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:16,997 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:16,997 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:16,997 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:16,997 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:16,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:16,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:16,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:16,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:16,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:16,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:16,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:16,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:16,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@283841e9[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:17,000 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:17,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:17,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:17,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:17,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:17,000 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:17,000 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,001 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:17,003 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:17,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:17,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:17,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:17,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:17,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:17,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:17,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:17,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:17,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:17,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:17,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:17,006 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:17,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:17,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:17,006 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:17,006 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:17,008 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:17,008 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:17,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:17,008 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:17,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:17,009 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,010 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,014 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:17,014 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:17,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:17,014 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,014 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:17,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:17,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:17,014 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:17,014 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:17,015 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,015 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:17,015 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:17,015 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:17,015 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:17,015 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:17,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:17,115 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:17,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:17,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:17,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:17,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:17,119 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:17,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:17,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:17,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:17,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:17,119 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:17,120 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:17,121 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:17,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:17,121 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,121 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:17,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:17,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:17,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:17,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:17,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:17,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:17,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:17,122 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:17,122 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:17,122 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:56:17,123 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:17,123 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:17,123 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:17,123 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:17,123 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:17,123 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:17,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:17,160 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:17,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397777182, max timestamp=1681397777191Num uniques 859; 2023-04-13 14:56:17,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:56:17,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397777182, max timestamp=1681397777191Num uniques 859; 2023-04-13 14:56:17,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:56:17,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397777182, max timestamp=1681397777191Num uniques 859; 2023-04-13 14:56:17,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:56:17,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:17,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397837222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:17,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397837225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:17,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397837256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:17,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397837258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:17,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397837326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:17,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397837329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:17,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397837529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:17,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397837532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,538 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304131fbf6dd0f9014a4c95c64f503b3f2503 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304131fbf6dd0f9014a4c95c64f503b3f2503 2023-04-13 14:56:17,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:17,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397837834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:17,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397837836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:17,945 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=327, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/9118a2ea6d1f439fac0ffe3dc2a98577 2023-04-13 14:56:18,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:18,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397838267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:18,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397838267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:18,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38362 deadline: 1681397838340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:18,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397838340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,355 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/5bd0284e6f2648e3956fecfb15a052df 2023-04-13 14:56:18,365 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/552bfdf7b4d144d9b8f5b3f7a5aa4860 2023-04-13 14:56:18,368 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/9118a2ea6d1f439fac0ffe3dc2a98577 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/9118a2ea6d1f439fac0ffe3dc2a98577 2023-04-13 14:56:18,372 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/9118a2ea6d1f439fac0ffe3dc2a98577, entries=150, sequenceid=327, filesize=25.4 K 2023-04-13 14:56:18,372 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/5bd0284e6f2648e3956fecfb15a052df as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/5bd0284e6f2648e3956fecfb15a052df 2023-04-13 14:56:18,376 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/5bd0284e6f2648e3956fecfb15a052df, entries=150, sequenceid=327, filesize=11.9 K 2023-04-13 14:56:18,376 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/552bfdf7b4d144d9b8f5b3f7a5aa4860 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/552bfdf7b4d144d9b8f5b3f7a5aa4860 2023-04-13 14:56:18,380 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/552bfdf7b4d144d9b8f5b3f7a5aa4860, entries=150, sequenceid=327, filesize=11.9 K 2023-04-13 14:56:18,381 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 1259ms, sequenceid=327, compaction requested=true 2023-04-13 14:56:18,381 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:18,381 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:18,381 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:18,381 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:18,381 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,381 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:18,381 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:18,386 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,386 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:18,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:18,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:18,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:18,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:18,387 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:18,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:18,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:18,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:18,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:18,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55a5506b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:18,389 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:18,389 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:18,390 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:18,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:18,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:18,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:18,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:18,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:18,391 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,391 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:18,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:18,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:18,392 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,392 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:18,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:18,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:18,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:18,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:18,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:18,394 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:18,394 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:18,394 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:18,394 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:18,394 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:18,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:18,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:18,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:18,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:18,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:18,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:18,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:18,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:18,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:18,396 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:18,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:18,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:18,400 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:18,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:18,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:18,403 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:18,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:18,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:18,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:18,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:18,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:18,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:18,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:18,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:18,404 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:18,404 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:18,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:18,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:18,504 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:18,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:18,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:18,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:18,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:18,507 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:18,507 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:18,507 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:18,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:18,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:18,508 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:18,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,509 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:18,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:18,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:18,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,510 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:18,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:18,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:18,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:18,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:18,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:18,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:18,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:18,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:18,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:18,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:18,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:18,511 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:18,511 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:18,511 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:18,511 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-13 14:56:18,512 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:18,512 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:18,512 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:18,512 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:18,512 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:18,512 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:18,530 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413c4cee690218a439f9dac77a694ff8474 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c4cee690218a439f9dac77a694ff8474 2023-04-13 14:56:18,535 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=363, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/ecffca786aa048e4a98b60f1b7b54fa6 2023-04-13 14:56:18,946 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/14631e34520a4d98aa3e84b4215b31c9 2023-04-13 14:56:18,961 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/7c79fd38e28545a98b5a76088580f85a 2023-04-13 14:56:18,964 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/ecffca786aa048e4a98b60f1b7b54fa6 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ecffca786aa048e4a98b60f1b7b54fa6 2023-04-13 14:56:18,967 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ecffca786aa048e4a98b60f1b7b54fa6, entries=150, sequenceid=363, filesize=25.4 K 2023-04-13 14:56:18,968 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/14631e34520a4d98aa3e84b4215b31c9 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/14631e34520a4d98aa3e84b4215b31c9 2023-04-13 14:56:18,971 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/14631e34520a4d98aa3e84b4215b31c9, entries=150, sequenceid=363, filesize=11.9 K 2023-04-13 14:56:18,972 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/7c79fd38e28545a98b5a76088580f85a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/7c79fd38e28545a98b5a76088580f85a 2023-04-13 14:56:18,982 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/7c79fd38e28545a98b5a76088580f85a, entries=150, sequenceid=363, filesize=11.9 K 2023-04-13 14:56:18,982 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 471ms, sequenceid=363, compaction requested=true 2023-04-13 14:56:18,982 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:18,983 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:18,983 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:18,983 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:18,983 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,983 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:18,983 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:18,984 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,984 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:18,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:18,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:18,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:18,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:18,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:18,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559117f3[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:18,986 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:18,986 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:18,991 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:18,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:18,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:18,991 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:18,991 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:18,991 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,991 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:18,991 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:18,992 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:18,992 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,992 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:18,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:18,995 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:18,996 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:18,996 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:18,996 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:18,996 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:18,997 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:18,997 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:18,997 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:18,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:18,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:18,998 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:18,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:19,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:19,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:19,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:19,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:19,002 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:19,002 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:19,003 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:19,003 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,003 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,003 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,003 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:19,003 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:19,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:19,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:19,103 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:19,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:19,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:19,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:19,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:19,106 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:19,107 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:19,107 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:19,107 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:19,107 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,107 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:19,108 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:19,108 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:19,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:19,109 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:19,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:19,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:19,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:19,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:19,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:19,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:19,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:19,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:19,110 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:19,110 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:19,111 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:19,111 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:19,111 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:19,112 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:19,113 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,113 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:19,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:19,114 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:19,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a560882[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:19,115 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,116 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:19,116 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,117 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:19,118 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,118 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:19,118 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:19,118 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:19,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:19,121 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:19,121 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:19,121 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:19,121 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:19,121 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:19,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:19,123 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:19,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:19,123 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:19,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,127 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:19,127 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:19,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:19,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:19,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:19,127 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,127 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:19,127 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:19,127 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:19,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:19,127 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:19,127 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,127 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:19,127 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,128 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:19,128 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,228 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:19,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:19,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:19,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:19,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:19,235 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:19,235 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:19,235 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:19,236 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:19,236 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,236 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:19,237 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:19,237 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:19,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:19,238 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,238 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:19,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:19,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:19,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:19,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:19,239 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:19,239 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:19,239 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:19,239 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:19,239 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:19,239 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:19,239 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:19,239 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:19,239 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:19,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:19,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:19,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:19,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:19,242 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,242 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:19,242 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:19,242 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:19,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@300b2d28[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:19,244 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,245 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,245 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:19,245 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:19,245 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:19,245 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,245 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:19,246 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:19,246 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,246 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:19,247 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:19,249 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:19,249 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:19,249 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:19,249 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:19,249 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:19,251 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,251 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:19,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,251 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:19,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:19,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:19,252 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:19,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:19,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:19,253 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,257 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,257 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:19,257 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:19,257 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:19,257 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:19,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:19,257 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,257 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:19,257 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:19,257 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:19,257 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:19,258 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:19,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:19,257 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:19,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:19,358 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:19,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:19,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:19,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:19,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:19,368 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:19,368 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:19,368 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:19,368 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:19,368 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,369 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:19,370 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:19,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:19,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:19,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:19,370 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:19,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:19,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:19,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:19,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:19,371 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:19,371 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:19,371 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:19,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:19,377 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:19,377 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:19,377 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:19,377 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 14:56:19,377 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:19,378 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:19,378 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:19,378 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:19,378 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:19,378 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:19,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:19,389 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:19,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681397779429, max timestamp=1681397779437Num uniques 859; 2023-04-13 14:56:19,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:56:19,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681397779429, max timestamp=1681397779437Num uniques 859; 2023-04-13 14:56:19,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:56:19,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681397779429, max timestamp=1681397779437Num uniques 859; 2023-04-13 14:56:19,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:56:19,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681397779456, max timestamp=1681397779478Num uniques 1068; 2023-04-13 14:56:19,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681397779456, max timestamp=1681397779478Num uniques 1068; 2023-04-13 14:56:19,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-13 14:56:19,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681397779456, max timestamp=1681397779478Num uniques 1068; 2023-04-13 14:56:19,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-13 14:56:19,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-13 14:56:19,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681397779505, max timestamp=1681397779518Num uniques 1077; 2023-04-13 14:56:19,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-13 14:56:19,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681397779505, max timestamp=1681397779518Num uniques 1077; 2023-04-13 14:56:19,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681397779505, max timestamp=1681397779518Num uniques 1077; 2023-04-13 14:56:19,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-13 14:56:19,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-13 14:56:19,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681397779550, max timestamp=1681397779559Num uniques 1036; 2023-04-13 14:56:19,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681397779550, max timestamp=1681397779559Num uniques 1036; 2023-04-13 14:56:19,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-13 14:56:19,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681397779550, max timestamp=1681397779559Num uniques 1036; 2023-04-13 14:56:19,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-13 14:56:19,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-13 14:56:19,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681397779584, max timestamp=1681397779597Num uniques 1045; 2023-04-13 14:56:19,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681397779584, max timestamp=1681397779597Num uniques 1045; 2023-04-13 14:56:19,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-13 14:56:19,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-13 14:56:19,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681397779584, max timestamp=1681397779597Num uniques 1045; 2023-04-13 14:56:19,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-13 14:56:19,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681397779616, max timestamp=1681397779637Num uniques 1054; 2023-04-13 14:56:19,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681397779616, max timestamp=1681397779637Num uniques 1054; 2023-04-13 14:56:19,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-13 14:56:19,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681397779616, max timestamp=1681397779637Num uniques 1054; 2023-04-13 14:56:19,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-13 14:56:19,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-13 14:56:19,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681397779669, max timestamp=1681397779678Num uniques 1063; 2023-04-13 14:56:19,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-13 14:56:19,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681397779669, max timestamp=1681397779678Num uniques 1063; 2023-04-13 14:56:19,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-13 14:56:19,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681397779669, max timestamp=1681397779678Num uniques 1063; 2023-04-13 14:56:19,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-13 14:56:19,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681397779712, max timestamp=1681397779733Num uniques 1022; 2023-04-13 14:56:19,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681397779712, max timestamp=1681397779733Num uniques 1022; 2023-04-13 14:56:19,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681397779712, max timestamp=1681397779733Num uniques 1022; 2023-04-13 14:56:19,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-13 14:56:19,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-13 14:56:19,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-13 14:56:19,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681397779755, max timestamp=1681397779779Num uniques 1081; 2023-04-13 14:56:19,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-13 14:56:19,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681397779755, max timestamp=1681397779779Num uniques 1081; 2023-04-13 14:56:19,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-13 14:56:19,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681397779766, max timestamp=1681397779779Num uniques 1081; 2023-04-13 14:56:19,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-13 14:56:19,815 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304138e05875ec4a94c1b99d689d344c9329d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304138e05875ec4a94c1b99d689d344c9329d 2023-04-13 14:56:19,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681397779804, max timestamp=1681397779825Num uniques 1090; 2023-04-13 14:56:19,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-13 14:56:19,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681397779804, max timestamp=1681397779825Num uniques 1090; 2023-04-13 14:56:19,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681397779804, max timestamp=1681397779825Num uniques 1040; 2023-04-13 14:56:19,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-13 14:56:19,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-13 14:56:19,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681397779879, max timestamp=1681397779889Num uniques 1049; 2023-04-13 14:56:19,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-13 14:56:19,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681397779879, max timestamp=1681397779889Num uniques 1049; 2023-04-13 14:56:19,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-13 14:56:19,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681397779879, max timestamp=1681397779889Num uniques 1049; 2023-04-13 14:56:19,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-13 14:56:19,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681397779921, max timestamp=1681397779929Num uniques 1058; 2023-04-13 14:56:19,932 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-13 14:56:19,932 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681397779921, max timestamp=1681397779929Num uniques 1058; 2023-04-13 14:56:19,932 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681397779921, max timestamp=1681397779929Num uniques 1058; 2023-04-13 14:56:19,932 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-13 14:56:19,932 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-13 14:56:19,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681397779943, max timestamp=1681397779963Num uniques 1067; 2023-04-13 14:56:19,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-13 14:56:19,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681397779943, max timestamp=1681397779963Num uniques 1067; 2023-04-13 14:56:19,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-13 14:56:19,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681397779957, max timestamp=1681397779963Num uniques 1067; 2023-04-13 14:56:19,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-13 14:56:20,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681397780000, max timestamp=1681397780009Num uniques 1076; 2023-04-13 14:56:20,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681397780000, max timestamp=1681397780009Num uniques 1026; 2023-04-13 14:56:20,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-13 14:56:20,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681397780000, max timestamp=1681397780009Num uniques 1076; 2023-04-13 14:56:20,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-13 14:56:20,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-13 14:56:20,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681397780025, max timestamp=1681397780052Num uniques 1035; 2023-04-13 14:56:20,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-13 14:56:20,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681397780025, max timestamp=1681397780052Num uniques 1035; 2023-04-13 14:56:20,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681397780025, max timestamp=1681397780052Num uniques 1035; 2023-04-13 14:56:20,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-13 14:56:20,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-13 14:56:20,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681397780086, max timestamp=1681397780111Num uniques 1044; 2023-04-13 14:56:20,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-13 14:56:20,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681397780086, max timestamp=1681397780111Num uniques 1044; 2023-04-13 14:56:20,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-13 14:56:20,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681397780086, max timestamp=1681397780111Num uniques 1044; 2023-04-13 14:56:20,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-13 14:56:20,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681397780130, max timestamp=1681397780157Num uniques 1053; 2023-04-13 14:56:20,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-13 14:56:20,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681397780130, max timestamp=1681397780157Num uniques 1053; 2023-04-13 14:56:20,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-13 14:56:20,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681397780130, max timestamp=1681397780157Num uniques 1053; 2023-04-13 14:56:20,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-13 14:56:20,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1681397780181, max timestamp=1681397780206Num uniques 1012; 2023-04-13 14:56:20,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1681397780181, max timestamp=1681397780206Num uniques 1062; 2023-04-13 14:56:20,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-13 14:56:20,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-04-13 14:56:20,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1681397780181, max timestamp=1681397780206Num uniques 1062; 2023-04-13 14:56:20,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-13 14:56:20,227 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=372, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/04a2fb821515475d970916a0910ccf6c 2023-04-13 14:56:20,237 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/bbd559cd43264193a97461cd308334a7 2023-04-13 14:56:20,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681397780244, max timestamp=1681397780256Num uniques 1021; 2023-04-13 14:56:20,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-13 14:56:20,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681397780244, max timestamp=1681397780256Num uniques 1021; 2023-04-13 14:56:20,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681397780244, max timestamp=1681397780256Num uniques 1021; 2023-04-13 14:56:20,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-13 14:56:20,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-13 14:56:20,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1681397780293, max timestamp=1681397780301Num uniques 1030; 2023-04-13 14:56:20,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681397780294, max timestamp=1681397780301Num uniques 1030; 2023-04-13 14:56:20,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-04-13 14:56:20,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-13 14:56:20,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8238, totalHeapSize=18960, min timestamp=1681397780294, max timestamp=1681397780301Num uniques 1030; 2023-04-13 14:56:20,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47168, new segment data size=8238, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-13 14:56:20,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1681397780315, max timestamp=1681397780324Num uniques 1039; 2023-04-13 14:56:20,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=13234, suffix heap size=136024,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=289 2023-04-13 14:56:20,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1681397780315, max timestamp=1681397780324Num uniques 1089; 2023-04-13 14:56:20,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=13234, suffix heap size=141224,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=289 2023-04-13 14:56:20,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1681397780315, max timestamp=1681397780324Num uniques 1039; 2023-04-13 14:56:20,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-13 14:56:20,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397840344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1681397780336, max timestamp=1681397780346Num uniques 1148; 2023-04-13 14:56:20,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=13648, suffix heap size=147360,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=298 2023-04-13 14:56:20,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397840344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1681397780336, max timestamp=1681397780346Num uniques 1148; 2023-04-13 14:56:20,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=13648, suffix heap size=147360,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=298 2023-04-13 14:56:20,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1681397780336, max timestamp=1681397780346Num uniques 1098; 2023-04-13 14:56:20,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=13648, suffix heap size=142160,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=298 2023-04-13 14:56:20,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 429 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397840386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397840391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397840450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397840450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397840492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397840498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38442 deadline: 1681397840601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,606 DEBUG [Thread-3819] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:56:20,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397840653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397840654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,674 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/9290b4f8ec004061933713b6ee03fbd3 2023-04-13 14:56:20,678 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/04a2fb821515475d970916a0910ccf6c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/04a2fb821515475d970916a0910ccf6c 2023-04-13 14:56:20,684 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/04a2fb821515475d970916a0910ccf6c, entries=200, sequenceid=372, filesize=32.3 K 2023-04-13 14:56:20,685 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/bbd559cd43264193a97461cd308334a7 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/bbd559cd43264193a97461cd308334a7 2023-04-13 14:56:20,688 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/bbd559cd43264193a97461cd308334a7, entries=100, sequenceid=372, filesize=9.5 K 2023-04-13 14:56:20,688 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/9290b4f8ec004061933713b6ee03fbd3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/9290b4f8ec004061933713b6ee03fbd3 2023-04-13 14:56:20,691 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/9290b4f8ec004061933713b6ee03fbd3, entries=100, sequenceid=372, filesize=9.5 K 2023-04-13 14:56:20,700 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=154.31 KB/158010 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 1323ms, sequenceid=372, compaction requested=true 2023-04-13 14:56:20,701 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:20,701 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:20,701 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:20,701 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:20,701 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,701 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:20,701 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:20,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1681397780377, max timestamp=1681397780698Num uniques 1157; 2023-04-13 14:56:20,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=7185, suffix heap size=148296,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=157 2023-04-13 14:56:20,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1681397780377, max timestamp=1681397780698Num uniques 1157; 2023-04-13 14:56:20,703 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=7185, suffix heap size=148296,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=157 2023-04-13 14:56:20,703 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:20,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:20,703 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=71.56 KB heapSize=195.06 KB 2023-04-13 14:56:20,703 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:20,704 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:20,704 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:20,704 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:20,704 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:20,704 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 14:56:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,706 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:20,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:20,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f52154a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:20,707 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:20,707 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,708 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,708 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:20,708 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:20,708 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:20,709 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,709 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:20,711 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,711 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:20,711 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:20,711 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:20,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:20,714 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:20,715 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:20,715 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:20,715 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:20,715 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:20,718 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,718 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:20,718 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:20,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,720 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,724 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,726 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:20,726 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:20,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:20,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:20,727 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,727 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:20,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:20,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:20,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:20,727 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:20,727 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:20,727 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:20,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:20,727 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:20,728 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,728 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,728 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,728 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,756 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413ba4dd3f6ac9041b9b0f3e307eb2b0ca0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ba4dd3f6ac9041b9b0f3e307eb2b0ca0 2023-04-13 14:56:20,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397840753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 442 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397840759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,828 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:20,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:20,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:20,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:20,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:20,831 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:20,831 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:20,831 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:20,831 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:20,832 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,832 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:20,833 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,833 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:20,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:20,833 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,833 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:20,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:20,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:20,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:20,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:20,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:20,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:20,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:20,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:20,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:20,835 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:20,835 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:20,835 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:20,835 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:20,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:20,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:20,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:20,835 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:20,838 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:20,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,838 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:20,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:20,839 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:20,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1221addc[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:20,839 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,841 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,841 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:20,841 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:20,841 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:20,841 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,841 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:20,842 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,842 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:20,842 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:20,842 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:20,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:20,845 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:20,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:20,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:20,845 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:20,845 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:20,846 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,846 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:20,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,846 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:20,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:20,847 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:20,848 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:20,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:20,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:20,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:20,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:20,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:20,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:20,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,851 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:20,851 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:20,851 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:20,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:20,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:20,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:20,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397840860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 444 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397840864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,951 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:20,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:20,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:20,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:20,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:20,955 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:20,955 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:20,955 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:20,955 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:20,956 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,956 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:20,957 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,957 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:20,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:20,957 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,957 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:20,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:20,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:20,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:20,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:20,959 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:20,959 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:20,959 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:20,959 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:20,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397840957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:20,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397840958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,960 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,960 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:20,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,961 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:20,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:20,962 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:20,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7859b502[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:20,962 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,963 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:20,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:20,964 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:20,964 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,964 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:20,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:20,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:20,966 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,966 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:20,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:20,969 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:20,969 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:20,969 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:20,969 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:20,969 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:20,970 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,970 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:20,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,970 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:20,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:20,971 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,972 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:20,974 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,975 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:20,975 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:20,975 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:20,975 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:20,975 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:20,975 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:20,975 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,975 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:20,975 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:20,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:20,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:20,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:20,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:20,975 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:20,975 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:20,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:21,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:21,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397841065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:21,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 446 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397841068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,075 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:21,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:21,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:21,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:21,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:21,078 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:21,078 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:21,078 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:21,078 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:21,079 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:21,079 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:21,080 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:21,080 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:21,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:21,080 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,080 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:21,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:21,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:21,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:21,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:21,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:21,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:21,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:21,081 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:21,081 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:21,081 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:21,081 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:21,081 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:21,081 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:21,082 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:21,082 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:21,082 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:21,082 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:21,082 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:21,082 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,082 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:21,082 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:21,087 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,087 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:21,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:21,089 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:21,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11d2a54c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:21,089 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:21,090 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:21,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:21,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:21,090 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:21,090 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:21,090 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,090 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:21,090 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:21,091 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,091 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:21,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:21,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:21,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:21,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:21,094 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:21,094 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:21,094 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:21,094 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:21,094 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:21,095 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:21,095 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:21,095 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:21,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:21,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:21,096 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:21,097 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,102 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,102 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:21,102 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:21,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:21,102 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:21,102 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:21,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:21,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:21,102 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:21,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:21,102 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,102 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:21,102 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:21,102 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:21,102 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:21,102 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:21,172 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=771, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/1f2914dc00bc4ce19d0db86977ddf8a3 2023-04-13 14:56:21,181 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=771 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/8665ac2ba4ad4ee394f0f7dcd545a9e8 2023-04-13 14:56:21,189 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=771 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/b6f493632f3b4ed0b134262ab99e8f4d 2023-04-13 14:56:21,193 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/1f2914dc00bc4ce19d0db86977ddf8a3 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/1f2914dc00bc4ce19d0db86977ddf8a3 2023-04-13 14:56:21,196 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/1f2914dc00bc4ce19d0db86977ddf8a3, entries=150, sequenceid=771, filesize=25.4 K 2023-04-13 14:56:21,197 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/8665ac2ba4ad4ee394f0f7dcd545a9e8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8665ac2ba4ad4ee394f0f7dcd545a9e8 2023-04-13 14:56:21,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8665ac2ba4ad4ee394f0f7dcd545a9e8, entries=150, sequenceid=771, filesize=11.9 K 2023-04-13 14:56:21,201 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/b6f493632f3b4ed0b134262ab99e8f4d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b6f493632f3b4ed0b134262ab99e8f4d 2023-04-13 14:56:21,203 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:21,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:21,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:21,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:21,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:21,211 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:21,211 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:21,212 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:21,212 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:21,212 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:21,212 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:21,213 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b6f493632f3b4ed0b134262ab99e8f4d, entries=150, sequenceid=771, filesize=11.9 K 2023-04-13 14:56:21,213 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:21,213 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:21,214 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~71.56 KB/73280, heapSize ~195.02 KB/199696, currentSize=107.34 KB/109920 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 511ms, sequenceid=771, compaction requested=true 2023-04-13 14:56:21,214 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:21,214 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,214 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:21,214 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:21,214 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 14:56:21,214 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 14:56:21,214 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:21,214 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:21,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:21,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:21,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:21,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:21,218 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 182402 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 14:56:21,218 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/A is initiating minor compaction (all files) 2023-04-13 14:56:21,218 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/A in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:21,218 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/207be9deabcc41ca8afb1ac5066be5e2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ce4e3a9b6bb14438b241c203712fb40c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/7f86114abf57477ca67b780a3dda4880, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/9118a2ea6d1f439fac0ffe3dc2a98577, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ecffca786aa048e4a98b60f1b7b54fa6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/04a2fb821515475d970916a0910ccf6c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/1f2914dc00bc4ce19d0db86977ddf8a3] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=178.1 K 2023-04-13 14:56:21,218 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 80506 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 14:56:21,218 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/B is initiating minor compaction (all files) 2023-04-13 14:56:21,218 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/B in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:21,218 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/1660bc04e00b4111b25cb2e2317cc7a5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/4a0c45f66df544099054ce033e05cb9d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c8ae2684a539459ea0c94ec7118a602c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/5bd0284e6f2648e3956fecfb15a052df, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/14631e34520a4d98aa3e84b4215b31c9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/bbd559cd43264193a97461cd308334a7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8665ac2ba4ad4ee394f0f7dcd545a9e8] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=78.6 K 2023-04-13 14:56:21,220 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 207be9deabcc41ca8afb1ac5066be5e2, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681397774284 2023-04-13 14:56:21,220 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 1660bc04e00b4111b25cb2e2317cc7a5, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681397774284 2023-04-13 14:56:21,221 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting ce4e3a9b6bb14438b241c203712fb40c, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681397774620 2023-04-13 14:56:21,221 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 4a0c45f66df544099054ce033e05cb9d, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681397774620 2023-04-13 14:56:21,222 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 7f86114abf57477ca67b780a3dda4880, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681397775032 2023-04-13 14:56:21,222 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting c8ae2684a539459ea0c94ec7118a602c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681397775032 2023-04-13 14:56:21,222 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 9118a2ea6d1f439fac0ffe3dc2a98577, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1681397776130 2023-04-13 14:56:21,222 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 5bd0284e6f2648e3956fecfb15a052df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1681397776130 2023-04-13 14:56:21,223 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting ecffca786aa048e4a98b60f1b7b54fa6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1681397777210 2023-04-13 14:56:21,223 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 14631e34520a4d98aa3e84b4215b31c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1681397777210 2023-04-13 14:56:21,223 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 04a2fb821515475d970916a0910ccf6c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681397779364 2023-04-13 14:56:21,223 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting bbd559cd43264193a97461cd308334a7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681397779364 2023-04-13 14:56:21,223 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 1f2914dc00bc4ce19d0db86977ddf8a3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1681397780377 2023-04-13 14:56:21,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:21,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:21,224 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 8665ac2ba4ad4ee394f0f7dcd545a9e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1681397780377 2023-04-13 14:56:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:21,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:21,240 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:21,240 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:21,240 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:21,240 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 14:56:21,240 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:21,240 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:21,240 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:21,241 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:21,241 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:21,241 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:21,247 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#B#compaction#565 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:21,254 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#A#compaction#566 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:21,263 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/369962ac82fb46b386fcb79e2cb64f59 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/369962ac82fb46b386fcb79e2cb64f59 2023-04-13 14:56:21,267 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/B of 7f6fba518b666b8072f5bc2fa5e4a5d5 into 369962ac82fb46b386fcb79e2cb64f59(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:21,267 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:21,267 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/B, priority=9, startTime=1681397781214; duration=0sec 2023-04-13 14:56:21,267 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:21,267 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 14:56:21,271 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 80506 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 14:56:21,272 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/C is initiating minor compaction (all files) 2023-04-13 14:56:21,272 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/C in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:21,272 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6e0ffb6e98504efab5f28f2cb5ef7248, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/90d612a26d5e455dba9b350014d5ad33, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/5bdacb8d7a3d44d6ad0133bcd21c448e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/552bfdf7b4d144d9b8f5b3f7a5aa4860, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/7c79fd38e28545a98b5a76088580f85a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/9290b4f8ec004061933713b6ee03fbd3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b6f493632f3b4ed0b134262ab99e8f4d] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=78.6 K 2023-04-13 14:56:21,272 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 6e0ffb6e98504efab5f28f2cb5ef7248, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681397774284 2023-04-13 14:56:21,272 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 90d612a26d5e455dba9b350014d5ad33, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681397774620 2023-04-13 14:56:21,273 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 5bdacb8d7a3d44d6ad0133bcd21c448e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681397775032 2023-04-13 14:56:21,273 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 552bfdf7b4d144d9b8f5b3f7a5aa4860, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1681397776130 2023-04-13 14:56:21,274 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 7c79fd38e28545a98b5a76088580f85a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1681397777210 2023-04-13 14:56:21,274 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 9290b4f8ec004061933713b6ee03fbd3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681397779364 2023-04-13 14:56:21,274 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting b6f493632f3b4ed0b134262ab99e8f4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1681397780377 2023-04-13 14:56:21,285 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#C#compaction#568 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:21,297 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/b818a76702f249279767dc3d5c42d615 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b818a76702f249279767dc3d5c42d615 2023-04-13 14:56:21,301 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/C of 7f6fba518b666b8072f5bc2fa5e4a5d5 into b818a76702f249279767dc3d5c42d615(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:21,301 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:21,302 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/C, priority=9, startTime=1681397781214; duration=0sec 2023-04-13 14:56:21,302 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:21,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:21,375 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:21,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:21,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397841389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:21,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 453 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397841390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:21,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397841461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:21,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397841463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:21,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397841492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:21,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 455 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397841493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,679 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413013244a3179a4ca2a839e1bf3fe8ffeb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413013244a3179a4ca2a839e1bf3fe8ffeb 2023-04-13 14:56:21,690 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/e60df33ca8c64d508d3bc8a276ab9536 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/e60df33ca8c64d508d3bc8a276ab9536 2023-04-13 14:56:21,694 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/A of 7f6fba518b666b8072f5bc2fa5e4a5d5 into e60df33ca8c64d508d3bc8a276ab9536(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:21,694 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:21,694 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/A, priority=9, startTime=1681397781214; duration=0sec 2023-04-13 14:56:21,694 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:21,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:21,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397841695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:21,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:21,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 457 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397841697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:22,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397842000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:22,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 459 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397842002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,085 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=790, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/a4dd7133d4e6424488ce9d74b1fa7d6f 2023-04-13 14:56:22,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:22,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397842468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:22,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397842470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,497 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=790 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/e7598e9baaf74b72a22dbc483a0fa14f 2023-04-13 14:56:22,507 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=790 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/1d4faf94de244f4580ef98f85f78dae9 2023-04-13 14:56:22,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:22,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397842506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:22,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 461 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397842507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,511 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/a4dd7133d4e6424488ce9d74b1fa7d6f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/a4dd7133d4e6424488ce9d74b1fa7d6f 2023-04-13 14:56:22,514 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/a4dd7133d4e6424488ce9d74b1fa7d6f, entries=150, sequenceid=790, filesize=25.4 K 2023-04-13 14:56:22,515 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/e7598e9baaf74b72a22dbc483a0fa14f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/e7598e9baaf74b72a22dbc483a0fa14f 2023-04-13 14:56:22,518 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/e7598e9baaf74b72a22dbc483a0fa14f, entries=150, sequenceid=790, filesize=11.9 K 2023-04-13 14:56:22,519 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/1d4faf94de244f4580ef98f85f78dae9 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/1d4faf94de244f4580ef98f85f78dae9 2023-04-13 14:56:22,532 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/1d4faf94de244f4580ef98f85f78dae9, entries=150, sequenceid=790, filesize=11.9 K 2023-04-13 14:56:22,533 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 1293ms, sequenceid=790, compaction requested=false 2023-04-13 14:56:22,533 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:22,533 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:22,533 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:22,533 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:22,533 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,533 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:22,533 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:22,535 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:22,535 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:22,535 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:22,536 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e751a79[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:22,538 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:22,539 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:22,540 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:22,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:22,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:22,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:22,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:22,541 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,541 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:22,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:22,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:22,542 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:22,542 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:22,547 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:22,547 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:22,547 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:22,547 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:22,547 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:22,550 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:22,550 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:22,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:22,550 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:22,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:22,550 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:22,551 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:22,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:22,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:22,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:22,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:22,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:22,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:22,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:22,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:22,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:22,554 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:22,554 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:22,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:22,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:22,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:22,554 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:22,554 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:22,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:22,655 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:22,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:22,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:22,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:22,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:22,658 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:22,658 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:22,658 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:22,658 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:22,658 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:22,658 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:22,659 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,659 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:22,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:22,660 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:22,660 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:22,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:22,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:22,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:22,661 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:22,661 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:22,661 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 14:56:22,662 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:22,662 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:22,662 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:22,662 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:22,662 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:22,662 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:22,682 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041305a547d479a9479e9f25ec2dc2ff07bf to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041305a547d479a9479e9f25ec2dc2ff07bf 2023-04-13 14:56:22,688 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=806, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/5ad6d17e50374a9ba7899c6e9ae59ef4 2023-04-13 14:56:23,102 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=806 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/6f56139f88d141d0b859bd2e89dded2a 2023-04-13 14:56:23,114 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=806 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/6cc46e00499c472a88fdd1c958d2916f 2023-04-13 14:56:23,119 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/5ad6d17e50374a9ba7899c6e9ae59ef4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5ad6d17e50374a9ba7899c6e9ae59ef4 2023-04-13 14:56:23,122 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5ad6d17e50374a9ba7899c6e9ae59ef4, entries=150, sequenceid=806, filesize=25.4 K 2023-04-13 14:56:23,123 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/6f56139f88d141d0b859bd2e89dded2a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/6f56139f88d141d0b859bd2e89dded2a 2023-04-13 14:56:23,126 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/6f56139f88d141d0b859bd2e89dded2a, entries=150, sequenceid=806, filesize=11.9 K 2023-04-13 14:56:23,127 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/6cc46e00499c472a88fdd1c958d2916f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6cc46e00499c472a88fdd1c958d2916f 2023-04-13 14:56:23,134 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6cc46e00499c472a88fdd1c958d2916f, entries=150, sequenceid=806, filesize=11.9 K 2023-04-13 14:56:23,134 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 473ms, sequenceid=806, compaction requested=true 2023-04-13 14:56:23,134 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:23,134 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:23,134 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:23,135 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:23,135 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,135 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:23,135 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:23,136 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,136 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,137 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,137 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:23,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:23,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51a51126[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:23,138 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:23,139 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,141 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,141 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:23,141 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:23,141 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,141 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:23,141 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:23,143 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,143 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,143 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:23,143 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:23,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:23,146 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:23,146 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:23,146 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:23,146 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:23,146 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:23,147 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,147 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,147 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:23,148 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,149 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:23,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:23,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:23,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,152 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:23,152 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:23,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:23,152 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:23,153 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:23,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:23,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:23,253 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:23,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:23,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:23,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:23,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:23,256 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:23,256 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:23,257 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:23,257 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:23,257 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,257 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,259 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,259 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:23,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:23,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:23,260 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,260 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:23,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:23,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:23,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:23,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:23,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:23,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:23,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:23,261 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:23,262 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:23,262 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:23,262 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:23,262 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:23,262 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:23,262 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,262 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:23,262 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:23,263 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,263 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,264 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,264 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:23,265 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:23,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64615fb5[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:23,265 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,266 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,266 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:23,266 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:23,266 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:23,266 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,266 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:23,268 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:23,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:23,270 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:23,271 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:23,271 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:23,271 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:23,271 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:23,274 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,274 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,274 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,274 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,275 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,280 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:23,280 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:23,280 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:23,280 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:23,280 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:23,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:23,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:23,281 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,281 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,281 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,281 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,281 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:23,281 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:23,281 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:23,381 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:23,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:23,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:23,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:23,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:23,385 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:23,385 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:23,385 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:23,385 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:23,385 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,385 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,387 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:23,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:23,387 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,387 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,387 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:23,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:23,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:23,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:23,389 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:23,389 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:23,389 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:23,389 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:23,389 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:23,389 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:23,389 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:23,389 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:23,389 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,389 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:23,389 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:23,390 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,390 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,391 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:23,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:23,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@601d7eb0[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:23,393 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:23,393 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,395 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,395 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:23,395 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:23,395 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:23,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:23,396 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,396 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:23,396 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,396 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:23,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:23,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:23,399 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:23,399 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:23,399 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:23,399 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:23,399 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:23,400 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,400 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,400 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:23,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,401 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,402 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:23,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:23,404 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:23,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:23,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:23,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:23,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:23,405 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:23,405 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:23,404 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,405 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,405 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:23,405 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:23,405 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,405 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,505 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:23,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:23,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:23,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:23,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:23,509 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:23,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:23,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:23,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:23,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,509 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,511 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:23,511 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:23,512 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,512 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:23,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:23,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:23,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:23,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:23,513 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:23,513 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:23,513 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:23,513 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:23,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:23,514 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:23,515 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,515 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,515 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,515 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:23,517 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:23,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65d0618[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:23,517 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,518 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,519 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:23,519 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:23,519 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:23,519 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,519 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:23,520 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:23,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:23,523 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:23,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:23,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:23,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:23,523 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:23,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,526 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:23,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,529 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,531 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:23,531 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:23,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,532 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:23,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:23,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:23,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:23,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,532 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:23,532 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:23,532 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:23,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:23,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:23,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:23,569 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:56:23,570 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:23,570 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:23,570 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:23,570 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:23,570 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:23,570 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:23,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:23,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397843611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397783599, max timestamp=1681397783611Num uniques 859; 2023-04-13 14:56:23,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:23,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397783599, max timestamp=1681397783611Num uniques 859; 2023-04-13 14:56:23,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:23,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397783599, max timestamp=1681397783611Num uniques 859; 2023-04-13 14:56:23,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:23,632 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:23,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:23,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:23,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:23,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:23,636 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:23,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:23,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:23,636 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:23,637 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,637 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,639 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:23,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:23,639 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,639 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,639 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:23,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:23,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:23,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:23,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:23,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:23,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:23,641 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:23,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,643 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,643 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,643 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:23,644 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:23,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f881bd4[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:23,644 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,646 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,646 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:23,646 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:23,646 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:23,646 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,646 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:23,647 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,647 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:23,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,648 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:23,648 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:23,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:23,650 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:23,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:23,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:23,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:23,650 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:23,651 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,651 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:23,651 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:23,652 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,653 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:23,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:23,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:23,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:23,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:23,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,656 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:23,656 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:23,656 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:23,657 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:23,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:23,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 491 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397843667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:23,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397843714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,757 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:23,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:23,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:23,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:23,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:23,760 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:23,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:23,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:23,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:23,761 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,761 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,762 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:23,763 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:23,763 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,763 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:23,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:23,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:23,764 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:23,764 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:23,764 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:23,765 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:23,765 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:23,765 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:23,765 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:23,765 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:23,765 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:23,765 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:23,765 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:23,765 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,765 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:23,765 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:23,769 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,769 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11fa7f67[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:23,771 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:23,771 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,772 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:23,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:23,772 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:23,772 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,772 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:23,773 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:23,773 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:23,773 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,774 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:23,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 493 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397843771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:23,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:23,776 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:23,777 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:23,777 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:23,777 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:23,777 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:23,779 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,779 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:23,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,779 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:23,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:23,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,780 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,781 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,783 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:23,783 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:23,783 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:23,783 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:23,784 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:23,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:23,784 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,784 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,784 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,784 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,784 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:23,784 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:23,784 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:23,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:23,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:23,884 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:23,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:23,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:23,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:23,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:23,888 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:23,888 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:23,888 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:23,888 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:23,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,890 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,890 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:23,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:23,890 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,890 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:23,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:23,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:23,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:23,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:23,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:23,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:23,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:23,892 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:23,892 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:23,892 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:23,892 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:23,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:23,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:23,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:23,892 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:23,894 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,894 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,894 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,894 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:23,896 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:23,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a997b16[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:23,896 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,898 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:23,898 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:23,899 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,899 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:23,900 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,900 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:23,900 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:23,900 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:23,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:23,903 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:23,903 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:23,903 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:23,903 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:23,903 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:23,904 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,904 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,904 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:23,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:23,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:23,905 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,906 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,909 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,909 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,909 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:23,909 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:23,909 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:23,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:23,909 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:23,909 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:23,910 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:23,910 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:23,910 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:23,910 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:23,910 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:23,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:23,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:23,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397843918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:23,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 495 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397843990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:23,994 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304136793eb4cca03448ba55b6140ee132194 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136793eb4cca03448ba55b6140ee132194 2023-04-13 14:56:24,010 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:24,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:24,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:24,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:24,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:24,018 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:24,018 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,018 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:24,018 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:24,019 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,019 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,021 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:24,021 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:24,021 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,021 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:24,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:24,022 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:24,022 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,022 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:24,022 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:24,022 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:24,023 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,023 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:24,023 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:24,023 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:24,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:24,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:24,023 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:24,024 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,025 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:24,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@749ad39b[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:24,026 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:24,026 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,027 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:24,027 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,027 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:24,029 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:24,029 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:24,029 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:24,029 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:24,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:24,031 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:24,031 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:24,031 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:24,031 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:24,031 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:24,034 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,034 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,034 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:24,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:24,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,035 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,036 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,038 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:24,038 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:24,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:24,038 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:24,038 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:24,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:24,038 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:24,039 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,039 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:24,039 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,039 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:24,039 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,039 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,039 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,139 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:24,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:24,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:24,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:24,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:24,143 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:24,143 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,143 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:24,143 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:24,143 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,143 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,144 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:24,144 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:24,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:24,145 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,145 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:24,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:24,146 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,146 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:24,146 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:24,146 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,147 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:24,147 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:24,147 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,147 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:24,147 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:24,148 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,149 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,149 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,149 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bf33067[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:24,151 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:24,151 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,156 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,156 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:24,156 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:24,156 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,156 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:24,156 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:24,157 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,157 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:24,157 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:24,157 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:24,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:24,161 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:24,161 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:24,161 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:24,161 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:24,161 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:24,162 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,162 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,163 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:24,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,163 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,165 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,167 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:24,167 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:24,168 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:24,168 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:24,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:24,168 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:24,168 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,168 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,168 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,168 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,168 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,168 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:24,168 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:24,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:24,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:24,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397844223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,268 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:24,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:24,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:24,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:24,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:24,272 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:24,272 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,272 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:24,272 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:24,272 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,272 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,273 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,273 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:24,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,274 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,274 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:24,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:24,275 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:24,275 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,275 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:24,276 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:24,276 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:24,276 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,276 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:24,276 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:24,276 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,276 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:24,276 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:24,276 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,276 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:24,276 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:24,278 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,278 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,278 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,278 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:24,279 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:24,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b07f39[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:24,279 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,281 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,281 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:24,281 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:24,281 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:24,283 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:24,283 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,283 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:24,283 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:24,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:24,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:24,284 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:24,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:24,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:24,284 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:24,284 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:24,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,285 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,287 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,289 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,291 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,291 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:24,291 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:24,291 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:24,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:24,291 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:24,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:24,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:24,291 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,292 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,292 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,292 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,292 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,292 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:24,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:24,292 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:24,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:24,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 497 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397844294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,392 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:24,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:24,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:24,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:24,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:24,395 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:24,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:24,395 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:24,396 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,396 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,397 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,397 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:24,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:24,397 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,397 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:24,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:24,398 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=820, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/d46905e9f4a44027af189b49cdf69529 2023-04-13 14:56:24,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:24,399 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:24,399 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,399 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:24,399 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:24,399 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:24,399 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,399 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:24,399 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:24,399 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:24,400 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:24,402 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,402 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,403 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,403 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:24,406 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:24,406 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5820f4d2[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:24,408 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,408 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:24,408 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:24,408 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:24,408 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,408 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:24,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,409 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,409 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:24,409 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:24,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:24,411 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:24,411 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:24,411 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:24,411 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:24,411 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:24,413 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,413 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,413 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:24,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,414 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,418 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:24,418 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:24,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:24,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:24,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:24,418 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:24,418 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:24,418 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:24,418 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:24,418 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,418 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:24,418 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:24,418 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,418 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,418 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:24,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397844474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,477 DEBUG [Thread-3823] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4133 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:56:24,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:24,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397844484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,487 DEBUG [Thread-3827] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:56:24,518 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:24,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:24,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:24,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:24,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:24,522 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:24,522 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,522 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:24,522 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:24,522 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,522 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,523 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:24,523 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:24,524 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,524 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:24,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:24,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:24,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:24,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:24,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:24,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:24,525 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,525 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:24,525 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:24,525 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:24,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:24,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:24,525 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:24,527 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:24,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a84e246[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:24,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:24,529 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,530 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:24,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:24,530 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:24,530 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,530 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:24,534 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,534 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,534 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:24,535 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:24,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:24,537 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:24,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:24,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:24,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:24,537 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:24,538 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,538 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,539 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:24,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:24,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:24,539 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,541 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,542 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,543 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,543 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,543 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,543 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,543 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,543 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:24,543 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:24,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:24,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:24,543 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:24,543 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:24,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:24,544 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:24,544 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:24,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:24,644 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:24,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:24,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:24,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:24,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:24,648 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:24,648 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,648 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:24,648 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:24,648 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,648 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,649 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:24,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:24,649 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:24,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:24,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:24,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:24,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:24,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:24,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:24,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:24,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:24,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:24,652 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:24,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:24,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:24,652 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:24,655 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,655 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,655 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,655 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:24,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28269579[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:24,657 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:24,657 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,658 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,658 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:24,658 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:24,658 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:24,658 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,658 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:24,659 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,660 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:24,660 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,660 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:24,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:24,663 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:24,663 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:24,663 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:24,663 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:24,663 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:24,665 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,665 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:24,665 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,666 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,667 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,670 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,670 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:24,670 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,670 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:24,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:24,671 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,671 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,671 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,671 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,671 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:24,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:24,671 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:24,671 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:24,670 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:24,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:24,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:24,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397844729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,771 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:24,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:24,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:24,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:24,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:24,776 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:24,776 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,776 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:24,776 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:24,777 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,777 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,778 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,778 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:24,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:24,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:24,778 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,778 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:24,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:24,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:24,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:24,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:24,784 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,784 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:24,784 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:24,784 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:24,786 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,786 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,786 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,786 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:24,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22e57ef3[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:24,787 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:24,787 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,789 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,789 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:24,789 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:24,789 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:24,789 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,789 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:24,791 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,791 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,791 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:24,791 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:24,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:24,793 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:24,793 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:24,793 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:24,793 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:24,793 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:24,795 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,795 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:24,795 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:24,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:24,796 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,798 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,800 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:24,800 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:24,800 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:24,800 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:24,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:24,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:24,800 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,800 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:24,800 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:24,800 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,801 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,801 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,801 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:24,801 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:24,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:24,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 499 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397844799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,822 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=820 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/38ae8a1d34ba4a3cbb78ca4f612d2e5e 2023-04-13 14:56:24,830 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=820 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/901c1bf76d63469092d32c4e60cfb7f7 2023-04-13 14:56:24,835 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/d46905e9f4a44027af189b49cdf69529 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/d46905e9f4a44027af189b49cdf69529 2023-04-13 14:56:24,838 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/d46905e9f4a44027af189b49cdf69529, entries=200, sequenceid=820, filesize=32.3 K 2023-04-13 14:56:24,839 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/38ae8a1d34ba4a3cbb78ca4f612d2e5e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/38ae8a1d34ba4a3cbb78ca4f612d2e5e 2023-04-13 14:56:24,842 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/38ae8a1d34ba4a3cbb78ca4f612d2e5e, entries=100, sequenceid=820, filesize=9.5 K 2023-04-13 14:56:24,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/901c1bf76d63469092d32c4e60cfb7f7 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/901c1bf76d63469092d32c4e60cfb7f7 2023-04-13 14:56:24,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/901c1bf76d63469092d32c4e60cfb7f7, entries=100, sequenceid=820, filesize=9.5 K 2023-04-13 14:56:24,846 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 1277ms, sequenceid=820, compaction requested=true 2023-04-13 14:56:24,846 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:24,846 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:24,846 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:56:24,846 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:56:24,846 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:24,846 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:24,847 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:56:24,847 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:56:24,847 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/A is initiating minor compaction (all files) 2023-04-13 14:56:24,847 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/B is initiating minor compaction (all files) 2023-04-13 14:56:24,847 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/A in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,847 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/B in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,847 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/e60df33ca8c64d508d3bc8a276ab9536, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/a4dd7133d4e6424488ce9d74b1fa7d6f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5ad6d17e50374a9ba7899c6e9ae59ef4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/d46905e9f4a44027af189b49cdf69529] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=109.2 K 2023-04-13 14:56:24,847 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/369962ac82fb46b386fcb79e2cb64f59, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/e7598e9baaf74b72a22dbc483a0fa14f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/6f56139f88d141d0b859bd2e89dded2a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/38ae8a1d34ba4a3cbb78ca4f612d2e5e] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=45.8 K 2023-04-13 14:56:24,848 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting e60df33ca8c64d508d3bc8a276ab9536, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1681397780377 2023-04-13 14:56:24,848 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 369962ac82fb46b386fcb79e2cb64f59, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1681397780377 2023-04-13 14:56:24,848 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting a4dd7133d4e6424488ce9d74b1fa7d6f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1681397780749 2023-04-13 14:56:24,848 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting e7598e9baaf74b72a22dbc483a0fa14f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1681397780749 2023-04-13 14:56:24,849 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 5ad6d17e50374a9ba7899c6e9ae59ef4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=806, earliestPutTs=1681397781382 2023-04-13 14:56:24,849 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 6f56139f88d141d0b859bd2e89dded2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=806, earliestPutTs=1681397781382 2023-04-13 14:56:24,849 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting d46905e9f4a44027af189b49cdf69529, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1681397783548 2023-04-13 14:56:24,849 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 38ae8a1d34ba4a3cbb78ca4f612d2e5e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1681397783568 2023-04-13 14:56:24,856 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#B#compaction#577 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:24,859 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#A#compaction#578 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:24,901 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:24,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:24,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:24,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:24,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:24,904 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:24,904 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,904 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:24,904 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:24,906 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,906 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,907 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,907 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:24,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:24,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,908 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:24,908 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:24,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:24,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:24,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:24,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:24,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:24,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:24,909 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:24,909 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:24,909 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:24,909 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:24,909 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:24,909 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:24,909 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:24,909 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-13 14:56:24,910 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:24,910 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:24,910 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:24,910 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:24,910 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:24,910 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:24,926 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413e1230515141445108c0f1e523201c1e8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e1230515141445108c0f1e523201c1e8 2023-04-13 14:56:25,269 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/f38144990cf34ff897aec9f37d9c41ae as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/f38144990cf34ff897aec9f37d9c41ae 2023-04-13 14:56:25,272 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/B of 7f6fba518b666b8072f5bc2fa5e4a5d5 into f38144990cf34ff897aec9f37d9c41ae(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:25,273 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:25,273 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/B, priority=12, startTime=1681397784846; duration=0sec 2023-04-13 14:56:25,273 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:25,273 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 14:56:25,274 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 14:56:25,274 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/C is initiating minor compaction (all files) 2023-04-13 14:56:25,275 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/C in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:25,275 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b818a76702f249279767dc3d5c42d615, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/1d4faf94de244f4580ef98f85f78dae9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6cc46e00499c472a88fdd1c958d2916f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/901c1bf76d63469092d32c4e60cfb7f7] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=45.8 K 2023-04-13 14:56:25,275 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting b818a76702f249279767dc3d5c42d615, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1681397780377 2023-04-13 14:56:25,275 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 1d4faf94de244f4580ef98f85f78dae9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1681397780749 2023-04-13 14:56:25,276 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 6cc46e00499c472a88fdd1c958d2916f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=806, earliestPutTs=1681397781382 2023-04-13 14:56:25,276 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 901c1bf76d63469092d32c4e60cfb7f7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1681397783568 2023-04-13 14:56:25,284 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#C#compaction#580 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:25,334 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=856, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/70a0a2f7d1ee491898ae6cb1039d96a1 2023-04-13 14:56:25,344 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=856 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/c0b54d356af0483ca7a405e1dc9fd5bc 2023-04-13 14:56:25,373 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=856 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/ac7bd67454574b4594906a2163482ff4 2023-04-13 14:56:25,377 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/70a0a2f7d1ee491898ae6cb1039d96a1 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70a0a2f7d1ee491898ae6cb1039d96a1 2023-04-13 14:56:25,380 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70a0a2f7d1ee491898ae6cb1039d96a1, entries=150, sequenceid=856, filesize=25.4 K 2023-04-13 14:56:25,381 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/c0b54d356af0483ca7a405e1dc9fd5bc as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c0b54d356af0483ca7a405e1dc9fd5bc 2023-04-13 14:56:25,384 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c0b54d356af0483ca7a405e1dc9fd5bc, entries=150, sequenceid=856, filesize=11.9 K 2023-04-13 14:56:25,385 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/ac7bd67454574b4594906a2163482ff4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/ac7bd67454574b4594906a2163482ff4 2023-04-13 14:56:25,387 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/ac7bd67454574b4594906a2163482ff4, entries=150, sequenceid=856, filesize=11.9 K 2023-04-13 14:56:25,388 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 479ms, sequenceid=856, compaction requested=false 2023-04-13 14:56:25,388 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:25,388 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:25,388 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:25,388 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,396 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:25,396 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:25,403 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:25,404 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:25,404 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,404 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:25,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:25,406 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:25,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dbc9819[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:25,406 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,407 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,407 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:25,407 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:25,407 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:25,408 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,408 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:25,409 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:25,409 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:25,409 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:25,409 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:25,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:25,412 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:25,412 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:25,412 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:25,412 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:25,412 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:25,414 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,414 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:25,414 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:25,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,416 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,423 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,423 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,423 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:25,423 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:25,423 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:25,423 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:25,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:25,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:25,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:25,424 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:25,423 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,424 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:25,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:25,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:25,424 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,424 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,424 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,524 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:25,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:25,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:25,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:25,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:25,528 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:25,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:25,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:25,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:25,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,531 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,531 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:25,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:25,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:25,531 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,532 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:25,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:25,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:25,533 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:25,533 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:25,533 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:25,533 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:25,533 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:25,533 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:25,533 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:25,534 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:25,534 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:25,534 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:25,534 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:25,534 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,534 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:25,534 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:25,535 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,535 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:25,535 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,536 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:25,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:25,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2077c33a[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:25,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:25,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,538 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,538 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:25,539 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:25,539 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,539 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:25,539 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:25,541 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,541 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:25,541 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:25,541 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:25,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:25,547 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:25,547 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:25,547 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:25,547 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:25,547 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:25,548 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,548 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:25,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,548 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:25,549 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:25,550 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:25,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,553 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,553 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:25,553 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:25,553 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:25,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:25,553 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:25,553 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,553 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,553 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,553 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,553 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:25,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:25,556 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:25,556 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:25,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:25,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:25,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:25,656 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:25,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:25,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:25,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:25,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:25,659 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:25,660 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:25,660 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:25,660 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:25,660 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,660 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,661 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:25,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:25,662 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,662 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,662 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:25,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:25,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:25,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:25,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:25,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:25,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:25,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:25,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:25,663 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:25,663 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:25,663 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:25,663 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:25,663 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:25,664 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:25,664 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,664 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:25,664 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:25,666 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,666 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:25,667 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,667 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:25,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:25,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c660710[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:25,668 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:25,668 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,670 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,670 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:25,670 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:25,670 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:25,670 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,670 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:25,671 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,671 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:25,671 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:25,671 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:25,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:25,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:25,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:25,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:25,674 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:25,674 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:25,674 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:25,674 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:25,674 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:25,675 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,675 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:25,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:25,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,675 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:25,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:25,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:25,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:25,676 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:25,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,677 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:25,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:25,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:25,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:25,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:25,680 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,681 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,681 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:25,681 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:25,680 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:25,681 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:25,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:25,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:25,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:25,687 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/70551d01b0b54fc49bcfbbebe63ca5b4 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70551d01b0b54fc49bcfbbebe63ca5b4 2023-04-13 14:56:25,691 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/A of 7f6fba518b666b8072f5bc2fa5e4a5d5 into 70551d01b0b54fc49bcfbbebe63ca5b4(size=26.1 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:25,691 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:25,691 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/A, priority=12, startTime=1681397784846; duration=0sec 2023-04-13 14:56:25,691 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:25,693 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/670e73343e3048a188f6ba00bbee1734 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/670e73343e3048a188f6ba00bbee1734 2023-04-13 14:56:25,697 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/C of 7f6fba518b666b8072f5bc2fa5e4a5d5 into 670e73343e3048a188f6ba00bbee1734(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:25,697 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:25,697 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/C, priority=12, startTime=1681397784846; duration=0sec 2023-04-13 14:56:25,698 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:25,781 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:25,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:25,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:25,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:25,785 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:56:25,786 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:25,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:25,786 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:25,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:25,786 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:25,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:25,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:25,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:25,792 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:25,792 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:25,792 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:25,792 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:25,793 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,793 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,794 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,794 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:25,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:25,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:25,795 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,795 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:25,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:25,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:25,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:25,808 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:25,808 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:25,808 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:25,813 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:25,813 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:25,813 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:25,813 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:25,813 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:25,813 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:25,814 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:25,814 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:25,814 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,816 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:25,816 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:25,817 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,817 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:25,818 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:25,818 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:25,820 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:25,820 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@357df489[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:25,821 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,821 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:25,821 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:25,821 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:25,821 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,821 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:25,823 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:25,823 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:25,823 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:25,823 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:25,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:25,827 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:25,827 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:25,827 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:25,827 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:25,827 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:25,830 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,830 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,830 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:25,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:25,832 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,834 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397785820, max timestamp=1681397785832Num uniques 859; 2023-04-13 14:56:25,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:25,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397785820, max timestamp=1681397785832Num uniques 859; 2023-04-13 14:56:25,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:25,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397785820, max timestamp=1681397785832Num uniques 859; 2023-04-13 14:56:25,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:25,837 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:25,837 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:25,837 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,837 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,837 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,837 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:25,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:25,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:25,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:25,838 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:25,838 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:25,837 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,838 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:25,838 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,837 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:25,838 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304133891192dc0bb4f8d8c8792599372c7cf to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133891192dc0bb4f8d8c8792599372c7cf 2023-04-13 14:56:25,844 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=871, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/4b20323d6f5744bcb66a4977e833b98b 2023-04-13 14:56:25,856 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=871 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/5a3dc4f6817844a5a19531f1cd40a613 2023-04-13 14:56:25,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:25,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 514 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397845869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:25,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397845873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,879 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=871 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/b84ad2945fd34c2786cee7aff41c3e60 2023-04-13 14:56:25,883 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/4b20323d6f5744bcb66a4977e833b98b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/4b20323d6f5744bcb66a4977e833b98b 2023-04-13 14:56:25,887 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/4b20323d6f5744bcb66a4977e833b98b, entries=150, sequenceid=871, filesize=25.4 K 2023-04-13 14:56:25,888 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/5a3dc4f6817844a5a19531f1cd40a613 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/5a3dc4f6817844a5a19531f1cd40a613 2023-04-13 14:56:25,891 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/5a3dc4f6817844a5a19531f1cd40a613, entries=150, sequenceid=871, filesize=11.9 K 2023-04-13 14:56:25,891 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/b84ad2945fd34c2786cee7aff41c3e60 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b84ad2945fd34c2786cee7aff41c3e60 2023-04-13 14:56:25,895 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b84ad2945fd34c2786cee7aff41c3e60, entries=150, sequenceid=871, filesize=11.9 K 2023-04-13 14:56:25,896 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 111ms, sequenceid=871, compaction requested=true 2023-04-13 14:56:25,896 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:25,896 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:56:25,896 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:25,896 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:25,896 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:25,896 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:56:25,897 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78854 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:56:25,897 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/A is initiating minor compaction (all files) 2023-04-13 14:56:25,897 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/A in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:25,897 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70551d01b0b54fc49bcfbbebe63ca5b4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70a0a2f7d1ee491898ae6cb1039d96a1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/4b20323d6f5744bcb66a4977e833b98b] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=77.0 K 2023-04-13 14:56:25,898 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:56:25,898 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/B is initiating minor compaction (all files) 2023-04-13 14:56:25,898 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/B in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:25,898 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/f38144990cf34ff897aec9f37d9c41ae, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c0b54d356af0483ca7a405e1dc9fd5bc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/5a3dc4f6817844a5a19531f1cd40a613] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=36.4 K 2023-04-13 14:56:25,899 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 70551d01b0b54fc49bcfbbebe63ca5b4, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1681397781382 2023-04-13 14:56:25,899 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting f38144990cf34ff897aec9f37d9c41ae, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1681397781382 2023-04-13 14:56:25,899 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 70a0a2f7d1ee491898ae6cb1039d96a1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=856, earliestPutTs=1681397783644 2023-04-13 14:56:25,900 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting c0b54d356af0483ca7a405e1dc9fd5bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=856, earliestPutTs=1681397783644 2023-04-13 14:56:25,901 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 5a3dc4f6817844a5a19531f1cd40a613, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=871, earliestPutTs=1681397785774 2023-04-13 14:56:25,901 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 4b20323d6f5744bcb66a4977e833b98b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=871, earliestPutTs=1681397785774 2023-04-13 14:56:25,909 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#B#compaction#587 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:25,934 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#A#compaction#586 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:25,938 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:25,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:25,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:25,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:25,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:25,942 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:25,942 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:25,942 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:25,942 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:25,943 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,943 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,949 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:25,949 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:25,950 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,950 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:25,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:25,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:25,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:25,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:25,951 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:25,951 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:25,951 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:25,951 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:25,951 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:25,951 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:25,951 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:25,951 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-13 14:56:25,952 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:25,952 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:25,952 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:25,952 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:25,952 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:25,952 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:25,966 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413d56dbd410c8748f4b0bfa94cb075b92a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d56dbd410c8748f4b0bfa94cb075b92a 2023-04-13 14:56:25,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:25,976 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:25,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:25,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397845987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:25,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:25,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 520 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397845988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:26,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397846091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:26,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 522 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397846091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:26,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397846295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:26,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 524 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397846296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,339 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/8690f4440b7a4fef90d2afe02c6d6ac7 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8690f4440b7a4fef90d2afe02c6d6ac7 2023-04-13 14:56:26,349 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/B of 7f6fba518b666b8072f5bc2fa5e4a5d5 into 8690f4440b7a4fef90d2afe02c6d6ac7(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:26,349 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:26,349 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/B, priority=13, startTime=1681397785896; duration=0sec 2023-04-13 14:56:26,349 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:26,349 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:56:26,350 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:56:26,351 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/C is initiating minor compaction (all files) 2023-04-13 14:56:26,351 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/C in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:26,351 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/670e73343e3048a188f6ba00bbee1734, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/ac7bd67454574b4594906a2163482ff4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b84ad2945fd34c2786cee7aff41c3e60] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=36.4 K 2023-04-13 14:56:26,351 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 670e73343e3048a188f6ba00bbee1734, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1681397781382 2023-04-13 14:56:26,352 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting ac7bd67454574b4594906a2163482ff4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=856, earliestPutTs=1681397783644 2023-04-13 14:56:26,352 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting b84ad2945fd34c2786cee7aff41c3e60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=871, earliestPutTs=1681397785774 2023-04-13 14:56:26,355 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/49762fa6c2a24dec8ff127cb11382a8f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/49762fa6c2a24dec8ff127cb11382a8f 2023-04-13 14:56:26,358 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#C#compaction#589 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:26,362 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/A of 7f6fba518b666b8072f5bc2fa5e4a5d5 into 49762fa6c2a24dec8ff127cb11382a8f(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:26,362 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:26,362 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/A, priority=13, startTime=1681397785896; duration=0sec 2023-04-13 14:56:26,362 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:26,366 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/be9a75a69963458ab29e5d47069af1d9 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/be9a75a69963458ab29e5d47069af1d9 2023-04-13 14:56:26,369 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/C of 7f6fba518b666b8072f5bc2fa5e4a5d5 into be9a75a69963458ab29e5d47069af1d9(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:26,369 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:26,369 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/C, priority=13, startTime=1681397785896; duration=0sec 2023-04-13 14:56:26,370 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:26,374 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=908, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/8db0ea6a4e4e45e6806e47bc600a434a 2023-04-13 14:56:26,385 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=908 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/1363a54274084d56895cad4e0323766c 2023-04-13 14:56:26,400 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=908 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/6bcc629b80b340028b508bf4cbb612a5 2023-04-13 14:56:26,403 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/8db0ea6a4e4e45e6806e47bc600a434a as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/8db0ea6a4e4e45e6806e47bc600a434a 2023-04-13 14:56:26,407 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/8db0ea6a4e4e45e6806e47bc600a434a, entries=150, sequenceid=908, filesize=25.4 K 2023-04-13 14:56:26,407 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/1363a54274084d56895cad4e0323766c as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/1363a54274084d56895cad4e0323766c 2023-04-13 14:56:26,410 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/1363a54274084d56895cad4e0323766c, entries=150, sequenceid=908, filesize=11.9 K 2023-04-13 14:56:26,411 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/6bcc629b80b340028b508bf4cbb612a5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6bcc629b80b340028b508bf4cbb612a5 2023-04-13 14:56:26,414 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6bcc629b80b340028b508bf4cbb612a5, entries=150, sequenceid=908, filesize=11.9 K 2023-04-13 14:56:26,415 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 464ms, sequenceid=908, compaction requested=false 2023-04-13 14:56:26,415 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:26,415 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:26,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,416 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:26,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:26,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:26,417 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:26,417 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:26,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:26,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:26,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:26,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:26,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:26,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c18de6f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:26,419 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:26,419 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:26,420 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:26,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:26,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:26,420 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:26,420 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:26,420 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:26,421 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,421 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:26,422 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:26,422 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:26,422 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:26,422 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:26,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:26,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:26,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:26,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:26,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:26,425 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:26,425 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:26,425 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:26,425 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:26,425 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:26,427 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:26,427 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:26,427 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:26,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:26,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:26,428 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:26,430 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:26,433 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:26,433 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:26,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:26,433 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:26,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:26,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:26,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:26,433 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,433 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:26,433 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:26,433 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:26,433 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,433 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:26,433 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:26,434 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:26,434 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:26,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:26,534 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:26,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:26,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:26,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:26,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:26,537 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:26,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:26,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:26,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:26,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:26,537 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:26,539 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:26,539 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:26,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:26,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:26,539 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,539 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:26,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:26,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:26,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:26,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:26,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:26,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:26,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:26,540 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:26,541 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:26,541 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:26,541 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:26,541 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 14:56:26,541 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:26,541 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:26,541 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:26,541 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:26,541 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:26,541 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:26,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:26,601 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:26,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397786619, max timestamp=1681397786627Num uniques 859; 2023-04-13 14:56:26,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397786619, max timestamp=1681397786627Num uniques 859; 2023-04-13 14:56:26,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681397786619, max timestamp=1681397786627Num uniques 859; 2023-04-13 14:56:26,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:26,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:56:26,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 14:56:26,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:26,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397846658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:26,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 543 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397846660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:26,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397846763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:26,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 545 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397846764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,957 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304133b0cd353c8b14ec28cdeedf81afd2a87 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133b0cd353c8b14ec28cdeedf81afd2a87 2023-04-13 14:56:26,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:26,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 547 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397846966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:26,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397846966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:26,970 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=921, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/2b05fc04189248d3b8344f0c8c214c8f 2023-04-13 14:56:27,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:27,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 549 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397847270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:27,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397847270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,381 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=921 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/c5e0e787068d4ff2b8c2013041e16fa8 2023-04-13 14:56:27,390 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=921 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/f2c6c14bf9ff4fc3929c10f2b1a0404b 2023-04-13 14:56:27,394 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/2b05fc04189248d3b8344f0c8c214c8f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2b05fc04189248d3b8344f0c8c214c8f 2023-04-13 14:56:27,397 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2b05fc04189248d3b8344f0c8c214c8f, entries=150, sequenceid=921, filesize=25.4 K 2023-04-13 14:56:27,398 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/c5e0e787068d4ff2b8c2013041e16fa8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c5e0e787068d4ff2b8c2013041e16fa8 2023-04-13 14:56:27,400 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c5e0e787068d4ff2b8c2013041e16fa8, entries=150, sequenceid=921, filesize=11.9 K 2023-04-13 14:56:27,401 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/f2c6c14bf9ff4fc3929c10f2b1a0404b as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/f2c6c14bf9ff4fc3929c10f2b1a0404b 2023-04-13 14:56:27,404 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/f2c6c14bf9ff4fc3929c10f2b1a0404b, entries=150, sequenceid=921, filesize=11.9 K 2023-04-13 14:56:27,404 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 863ms, sequenceid=921, compaction requested=true 2023-04-13 14:56:27,404 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:27,404 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:27,404 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:27,406 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,406 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,406 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,407 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:27,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:27,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3268a6c5[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:27,408 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:27,408 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,411 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,411 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:27,411 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:27,411 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:27,411 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,411 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:27,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:27,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:27,413 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,413 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:27,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:27,415 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:27,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:27,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:27,415 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:27,415 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:27,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:27,416 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:27,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:27,417 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,418 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:27,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,421 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:27,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:27,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:27,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:27,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:27,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,421 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,421 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:27,421 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:27,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:27,521 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:27,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:27,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:27,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:27,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:27,525 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:27,525 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:27,525 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:27,525 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:27,525 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,525 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,527 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:27,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:27,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,528 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:27,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:27,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:27,529 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:27,529 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:27,530 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=131.94 KB heapSize=375.22 KB 2023-04-13 14:56:27,530 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:27,530 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:27,530 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:27,530 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:27,530 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:27,530 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:27,541 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304134402d421809e473d9f0c60585f8f34d8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134402d421809e473d9f0c60585f8f34d8 2023-04-13 14:56:27,546 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=957, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/2e7497d872d44f6192c704ab37165071 2023-04-13 14:56:27,576 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=957 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/f0a3d724bd9d4560903ced888e4289a5 2023-04-13 14:56:27,601 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=957 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/460d1faa61c64dddb8ad5e0faeb9e87e 2023-04-13 14:56:27,606 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/2e7497d872d44f6192c704ab37165071 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2e7497d872d44f6192c704ab37165071 2023-04-13 14:56:27,610 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2e7497d872d44f6192c704ab37165071, entries=150, sequenceid=957, filesize=25.4 K 2023-04-13 14:56:27,611 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/f0a3d724bd9d4560903ced888e4289a5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/f0a3d724bd9d4560903ced888e4289a5 2023-04-13 14:56:27,614 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/f0a3d724bd9d4560903ced888e4289a5, entries=150, sequenceid=957, filesize=11.9 K 2023-04-13 14:56:27,614 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/460d1faa61c64dddb8ad5e0faeb9e87e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/460d1faa61c64dddb8ad5e0faeb9e87e 2023-04-13 14:56:27,625 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/460d1faa61c64dddb8ad5e0faeb9e87e, entries=150, sequenceid=957, filesize=11.9 K 2023-04-13 14:56:27,626 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~375.17 KB/384176, currentSize=0 B/0 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 97ms, sequenceid=957, compaction requested=true 2023-04-13 14:56:27,626 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:27,626 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:27,626 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:27,626 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:27,626 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,626 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:27,626 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:27,627 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,628 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:27,628 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:27,628 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:27,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:27,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e3a791f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:27,630 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:27,630 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,631 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,631 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:27,631 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:27,631 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,631 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:27,631 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:27,632 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:27,632 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:27,633 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:27,633 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:27,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:27,635 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:27,635 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:27,635 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:27,635 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:27,635 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:27,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:27,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,638 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:27,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:27,639 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,640 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:27,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:27,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:27,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:27,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:27,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:27,643 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:27,643 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:27,642 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:27,642 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:27,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:27,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:27,643 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,743 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:27,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:27,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:27,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:27,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:27,746 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:27,747 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:27,747 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:27,747 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:27,747 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,747 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,751 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:27,751 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,752 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,752 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:27,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:27,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:27,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:27,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:27,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:27,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:27,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:27,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:27,753 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:27,753 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:27,753 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:27,753 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:27,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:27,753 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:27,754 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,754 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:27,754 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:27,756 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,756 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:27,757 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,757 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:27,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:27,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64391bed[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:27,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:27,758 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,760 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,760 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:27,760 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:27,760 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:27,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,760 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:27,761 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:27,761 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:27,761 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:27,761 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:27,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:27,764 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:27,764 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:27,764 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:27,764 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:27,764 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:27,766 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,766 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:27,766 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:27,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:27,767 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,768 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,771 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:27,771 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:27,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:27,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:27,771 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:27,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:27,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:27,771 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:27,771 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,771 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,771 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:27,771 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,772 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,772 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,772 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:27,772 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:27,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:27,813 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 14:56:27,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:27,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:27,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:27,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:27,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:27,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:27,828 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413cdae2be7c42743f3ad20289489c47527 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413cdae2be7c42743f3ad20289489c47527 2023-04-13 14:56:27,836 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=969, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/5bbc4dc88ba142858696b346d2bf5f4f 2023-04-13 14:56:27,845 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=969 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/beb9680ab6614d32976e3c28879ef3ec 2023-04-13 14:56:27,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:27,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397847852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,856 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397787836, max timestamp=1681397787854Num uniques 859; 2023-04-13 14:56:27,856 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:27,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397787836, max timestamp=1681397787854Num uniques 859; 2023-04-13 14:56:27,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681397787836, max timestamp=1681397787854Num uniques 859; 2023-04-13 14:56:27,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:27,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 14:56:27,872 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:27,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:27,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:27,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:27,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:27,875 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:27,875 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:27,875 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:27,875 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:27,875 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,875 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,876 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:27,876 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:27,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:27,877 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,877 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:27,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:27,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:27,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:27,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:27,878 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:27,878 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:27,878 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:27,879 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:27,879 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:27,879 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:27,879 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:27,880 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:27,881 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:27,881 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:27,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:27,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e384aa3[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:27,882 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:27,882 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,883 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,884 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:27,884 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:27,884 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,884 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:27,884 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:27,886 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,886 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:27,886 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:27,886 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:27,889 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:27,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:27,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:27,889 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:27,889 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:27,890 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:27,891 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,891 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:27,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:27,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,891 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,893 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:27,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:27,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:27,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:27,895 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,895 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:27,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:27,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:27,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:27,896 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:27,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:27,896 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:27,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:27,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 579 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397847929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:27,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397847957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:27,996 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:27,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:27,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:27,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:27,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:27,999 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:27,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:27,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:27,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:27,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:27,999 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,001 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,001 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:28,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:28,001 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,001 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:28,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:28,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:28,002 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:28,002 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:28,002 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:28,002 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:28,002 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:28,002 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:28,002 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:28,003 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:28,003 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:28,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:28,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:28,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:28,003 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:28,005 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,005 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:28,006 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,006 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:28,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:28,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79119b2f[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:28,007 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:28,007 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,008 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,008 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:28,008 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,008 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:28,008 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:28,008 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:28,010 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,010 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:28,010 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:28,010 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:28,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:28,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:28,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:28,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:28,013 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:28,013 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:28,013 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:28,013 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:28,013 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:28,015 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,015 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:28,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,015 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:28,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:28,016 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,017 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:28,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,020 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:28,020 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:28,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:28,020 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:28,020 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,021 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,021 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,021 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:28,021 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,021 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:28,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:28,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:28,021 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:28,021 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:28,021 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:28,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:28,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 581 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397848033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,121 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:28,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:28,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:28,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:28,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:28,125 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:28,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:28,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:28,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:28,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,125 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,127 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:28,127 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,127 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,127 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:28,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:28,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:28,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:28,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:28,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:28,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:28,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:28,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:28,128 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:28,129 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:28,129 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:28,129 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:28,129 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:28,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:28,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:28,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:28,129 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:28,130 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:28,130 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,131 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,131 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:28,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:28,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:28,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:28,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@140ee793[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:28,133 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:28,133 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,138 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,138 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:28,138 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:28,138 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,138 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:28,138 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:28,140 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,140 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:28,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:28,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:28,140 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:28,140 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:28,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:28,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:28,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:28,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:28,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:28,143 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:28,143 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:28,143 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:28,143 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:28,143 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:28,145 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,145 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:28,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,146 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:28,146 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,147 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:28,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:28,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:28,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:28,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:28,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:28,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,150 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,151 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,151 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,151 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,151 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:28,150 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:28,151 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:28,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:28,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:28,151 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,151 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:28,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 421 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397848162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:28,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 583 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397848238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,251 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:28,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:28,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:28,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:28,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:28,254 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:28,254 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:28,254 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:28,254 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:28,255 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,255 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,257 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:28,257 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:28,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,258 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:28,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:28,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:28,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:28,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:28,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:28,259 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=969 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/5a6de4b94c7746cf9e8173afc4df7700 2023-04-13 14:56:28,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:28,259 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:28,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:28,259 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:28,259 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:28,259 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:28,259 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:28,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:28,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:28,260 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:28,261 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,261 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:28,262 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,262 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/5bbc4dc88ba142858696b346d2bf5f4f as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5bbc4dc88ba142858696b346d2bf5f4f 2023-04-13 14:56:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:28,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:28,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c4608c[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:28,264 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:28,264 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,265 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,265 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:28,265 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:28,265 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:28,266 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,266 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:28,267 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5bbc4dc88ba142858696b346d2bf5f4f, entries=350, sequenceid=969, filesize=52.9 K 2023-04-13 14:56:28,267 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:28,267 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:28,267 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:28,267 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:28,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/beb9680ab6614d32976e3c28879ef3ec as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/beb9680ab6614d32976e3c28879ef3ec 2023-04-13 14:56:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:28,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:28,270 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:28,270 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:28,270 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:28,270 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:28,270 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:28,271 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/beb9680ab6614d32976e3c28879ef3ec, entries=150, sequenceid=969, filesize=11.9 K 2023-04-13 14:56:28,272 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/5a6de4b94c7746cf9e8173afc4df7700 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/5a6de4b94c7746cf9e8173afc4df7700 2023-04-13 14:56:28,273 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,273 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,273 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:28,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:28,274 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:28,275 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:28,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/5a6de4b94c7746cf9e8173afc4df7700, entries=150, sequenceid=969, filesize=11.9 K 2023-04-13 14:56:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:28,276 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 463ms, sequenceid=969, compaction requested=true 2023-04-13 14:56:28,276 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:28,277 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 14:56:28,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:28,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:28,277 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:28,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 14:56:28,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:28,277 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:28,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:28,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:28,277 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 14:56:28,277 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:28,277 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:28,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:28,277 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:28,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:28,277 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:28,277 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,278 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,278 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,278 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,278 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:28,278 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,278 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 159216 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 14:56:28,279 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/A is initiating minor compaction (all files) 2023-04-13 14:56:28,279 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/A in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:28,279 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/49762fa6c2a24dec8ff127cb11382a8f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/8db0ea6a4e4e45e6806e47bc600a434a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2b05fc04189248d3b8344f0c8c214c8f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2e7497d872d44f6192c704ab37165071, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5bbc4dc88ba142858696b346d2bf5f4f] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=155.5 K 2023-04-13 14:56:28,279 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 49762fa6c2a24dec8ff127cb11382a8f, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=871, earliestPutTs=1681397785774 2023-04-13 14:56:28,279 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 14:56:28,280 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 8db0ea6a4e4e45e6806e47bc600a434a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=908, earliestPutTs=1681397785857 2023-04-13 14:56:28,280 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/B is initiating minor compaction (all files) 2023-04-13 14:56:28,280 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/B in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:28,280 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8690f4440b7a4fef90d2afe02c6d6ac7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/1363a54274084d56895cad4e0323766c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c5e0e787068d4ff2b8c2013041e16fa8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/f0a3d724bd9d4560903ced888e4289a5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/beb9680ab6614d32976e3c28879ef3ec] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=60.3 K 2023-04-13 14:56:28,280 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 2b05fc04189248d3b8344f0c8c214c8f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=921, earliestPutTs=1681397785978 2023-04-13 14:56:28,280 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 8690f4440b7a4fef90d2afe02c6d6ac7, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=871, earliestPutTs=1681397785774 2023-04-13 14:56:28,280 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 2e7497d872d44f6192c704ab37165071, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=957, earliestPutTs=1681397786649 2023-04-13 14:56:28,280 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 1363a54274084d56895cad4e0323766c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=908, earliestPutTs=1681397785857 2023-04-13 14:56:28,281 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 5bbc4dc88ba142858696b346d2bf5f4f, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=969, earliestPutTs=1681397787792 2023-04-13 14:56:28,281 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting c5e0e787068d4ff2b8c2013041e16fa8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=921, earliestPutTs=1681397785978 2023-04-13 14:56:28,281 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting f0a3d724bd9d4560903ced888e4289a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=957, earliestPutTs=1681397786649 2023-04-13 14:56:28,281 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting beb9680ab6614d32976e3c28879ef3ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=969, earliestPutTs=1681397787795 2023-04-13 14:56:28,296 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#B#compaction#602 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:28,297 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#A#compaction#601 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:28,315 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/d48ab40c4b5a498ab66cb1c620ca9b04 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/d48ab40c4b5a498ab66cb1c620ca9b04 2023-04-13 14:56:28,318 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/B of 7f6fba518b666b8072f5bc2fa5e4a5d5 into d48ab40c4b5a498ab66cb1c620ca9b04(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:28,318 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:28,318 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/B, priority=11, startTime=1681397788277; duration=0sec 2023-04-13 14:56:28,319 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:28,319 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 14:56:28,320 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 14:56:28,320 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/C is initiating minor compaction (all files) 2023-04-13 14:56:28,320 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/C in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:28,320 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/be9a75a69963458ab29e5d47069af1d9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6bcc629b80b340028b508bf4cbb612a5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/f2c6c14bf9ff4fc3929c10f2b1a0404b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/460d1faa61c64dddb8ad5e0faeb9e87e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/5a6de4b94c7746cf9e8173afc4df7700] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=60.3 K 2023-04-13 14:56:28,320 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting be9a75a69963458ab29e5d47069af1d9, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=871, earliestPutTs=1681397785774 2023-04-13 14:56:28,321 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 6bcc629b80b340028b508bf4cbb612a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=908, earliestPutTs=1681397785857 2023-04-13 14:56:28,321 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting f2c6c14bf9ff4fc3929c10f2b1a0404b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=921, earliestPutTs=1681397785978 2023-04-13 14:56:28,321 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 460d1faa61c64dddb8ad5e0faeb9e87e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=957, earliestPutTs=1681397786649 2023-04-13 14:56:28,322 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 5a6de4b94c7746cf9e8173afc4df7700, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=969, earliestPutTs=1681397787795 2023-04-13 14:56:28,328 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#C#compaction#603 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:28,336 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/40f82a8b2b484663a42b4737c1464248 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/40f82a8b2b484663a42b4737c1464248 2023-04-13 14:56:28,339 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/C of 7f6fba518b666b8072f5bc2fa5e4a5d5 into 40f82a8b2b484663a42b4737c1464248(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:28,339 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:28,339 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/C, priority=11, startTime=1681397788277; duration=0sec 2023-04-13 14:56:28,339 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:28,378 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:28,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:28,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 14:56:28,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 14:56:28,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 14:56:28,381 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 14:56:28,381 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:28,381 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 14:56:28,381 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 14:56:28,381 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,381 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,382 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:28,382 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:28,383 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,383 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 14:56:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 14:56:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:28,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 14:56:28,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 14:56:28,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 14:56:28,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 14:56:28,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 14:56:28,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 14:56:28,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 14:56:28,384 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 14:56:28,384 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:28,384 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. started... 2023-04-13 14:56:28,384 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-13 14:56:28,384 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:28,385 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:28,385 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:28,385 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:28,385 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:28,385 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 14:56:28,404 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041330b4ba1e8bb640f7a3f7493aaef6511e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041330b4ba1e8bb640f7a3f7493aaef6511e 2023-04-13 14:56:28,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:28,481 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. as already flushing 2023-04-13 14:56:28,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:28,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 429 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397848501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:28,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397848503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:28,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1681397848517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,521 DEBUG [Thread-3823] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8177 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., hostname=jenkins-hbase4.apache.org,43875,1681397617638, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 14:56:28,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:28,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43875] ipc.CallRunner(144): callId: 585 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397848544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:28,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397848605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:28,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43875] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397848607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,694 DEBUG [Thread-3832] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c53bf96 to 127.0.0.1:62626 2023-04-13 14:56:28,694 DEBUG [Thread-3832] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:28,695 DEBUG [Thread-3830] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25baee3e to 127.0.0.1:62626 2023-04-13 14:56:28,695 DEBUG [Thread-3830] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:28,696 DEBUG [Thread-3836] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d765193 to 127.0.0.1:62626 2023-04-13 14:56:28,696 DEBUG [Thread-3836] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:28,697 DEBUG [Thread-3838] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32bf16cd to 127.0.0.1:62626 2023-04-13 14:56:28,698 DEBUG [Thread-3838] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:28,699 DEBUG [Thread-3834] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b59bdcf to 127.0.0.1:62626 2023-04-13 14:56:28,699 DEBUG [Thread-3834] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:28,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:28,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 433 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397848808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:28,809 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1007, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/e395780fb82347958cf3716ba4d79d50 2023-04-13 14:56:28,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:28,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397848810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:29,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 587 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38448 deadline: 1681397849050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:29,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397849111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:29,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397849112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,218 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1007 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/8edfc47949c4431a8fec96787fa4fc6e 2023-04-13 14:56:29,522 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/177a2fa19bf340388d7b009ad556e017 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/177a2fa19bf340388d7b009ad556e017 2023-04-13 14:56:29,525 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/A of 7f6fba518b666b8072f5bc2fa5e4a5d5 into 177a2fa19bf340388d7b009ad556e017(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-13 14:56:29,525 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:29,525 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/A, priority=11, startTime=1681397788277; duration=1sec 2023-04-13 14:56:29,525 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:29,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:29,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38362 deadline: 1681397849614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 14:56:29,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38370 deadline: 1681397849615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,625 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1007 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/87bfb321933649e38f7e12006d22ee04 2023-04-13 14:56:29,629 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/e395780fb82347958cf3716ba4d79d50 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/e395780fb82347958cf3716ba4d79d50 2023-04-13 14:56:29,631 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/e395780fb82347958cf3716ba4d79d50, entries=150, sequenceid=1007, filesize=25.4 K 2023-04-13 14:56:29,632 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/8edfc47949c4431a8fec96787fa4fc6e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8edfc47949c4431a8fec96787fa4fc6e 2023-04-13 14:56:29,635 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8edfc47949c4431a8fec96787fa4fc6e, entries=150, sequenceid=1007, filesize=11.9 K 2023-04-13 14:56:29,635 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/87bfb321933649e38f7e12006d22ee04 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/87bfb321933649e38f7e12006d22ee04 2023-04-13 14:56:29,638 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/87bfb321933649e38f7e12006d22ee04, entries=150, sequenceid=1007, filesize=11.9 K 2023-04-13 14:56:29,639 INFO [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 1255ms, sequenceid=1007, compaction requested=false 2023-04-13 14:56:29,639 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:29,639 DEBUG [rs(jenkins-hbase4.apache.org,43875,1681397617638)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:29,639 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 14:56:29,639 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 14:56:29,639 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,639 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 14:56:29,639 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 14:56:29,641 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:29,641 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:29,642 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 14:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43875,1681397617638' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 14:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74654865[Count = 0] remaining members to acquire global barrier 2023-04-13 14:56:29,643 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 14:56:29,643 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:29,644 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:29,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 14:56:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 14:56:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43875,1681397617638' in zk 2023-04-13 14:56:29,644 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,644 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 14:56:29,645 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 14:56:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,43875,1681397617638' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 14:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43875,1681397617638': 2023-04-13 14:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43875,1681397617638' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 14:56:29,647 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 14:56:29,647 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 14:56:29,647 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 14:56:29,647 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 14:56:29,647 INFO [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 14:56:29,649 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:29,649 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:29,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 14:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 14:56:29,649 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 14:56:29,649 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 14:56:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:29,650 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 14:56:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 14:56:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,653 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,653 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 14:56:29,653 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:29,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 14:56:29,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 14:56:29,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 14:56:29,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 14:56:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 14:56:29,653 DEBUG [(jenkins-hbase4.apache.org,46829,1681397616434)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 14:56:29,653 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 14:56:29,653 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 14:56:29,654 DEBUG [Thread-3829] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 14:56:29,654 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:29,654 DEBUG [Thread-3829] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 14:56:29,654 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:29,654 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 14:56:29,654 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 14:56:29,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 14:56:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 14:56:29,754 DEBUG [Thread-3829] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 14:56:29,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 14:56:30,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43875] regionserver.HRegion(9158): Flush requested on 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:30,054 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 14:56:30,054 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:30,054 DEBUG [Thread-3821] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f06dd50 to 127.0.0.1:62626 2023-04-13 14:56:30,054 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:30,054 DEBUG [Thread-3821] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:30,054 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:30,054 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:30,054 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:30,054 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:30,464 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413cbb0b9fb8cf149eead353424b1b22ef4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413cbb0b9fb8cf149eead353424b1b22ef4 2023-04-13 14:56:30,618 DEBUG [Thread-3827] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08b57aae to 127.0.0.1:62626 2023-04-13 14:56:30,618 DEBUG [Thread-3827] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:30,623 DEBUG [Thread-3825] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4db22237 to 127.0.0.1:62626 2023-04-13 14:56:30,623 DEBUG [Thread-3825] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:30,641 DEBUG [Thread-3819] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e3503b4 to 127.0.0.1:62626 2023-04-13 14:56:30,641 DEBUG [Thread-3819] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:30,868 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1020, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/110a8e2d146b49a3b2b55373d44fea58 2023-04-13 14:56:31,276 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1020 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/68f3bb84c7ce4ea485ea325013a91e1e 2023-04-13 14:56:31,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1020 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/7ffd83d408874ef68ff40dc90b709929 2023-04-13 14:56:31,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/110a8e2d146b49a3b2b55373d44fea58 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/110a8e2d146b49a3b2b55373d44fea58 2023-04-13 14:56:31,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/110a8e2d146b49a3b2b55373d44fea58, entries=150, sequenceid=1020, filesize=25.4 K 2023-04-13 14:56:31,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/68f3bb84c7ce4ea485ea325013a91e1e as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/68f3bb84c7ce4ea485ea325013a91e1e 2023-04-13 14:56:31,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/68f3bb84c7ce4ea485ea325013a91e1e, entries=150, sequenceid=1020, filesize=11.9 K 2023-04-13 14:56:31,693 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/7ffd83d408874ef68ff40dc90b709929 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/7ffd83d408874ef68ff40dc90b709929 2023-04-13 14:56:31,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/7ffd83d408874ef68ff40dc90b709929, entries=150, sequenceid=1020, filesize=11.9 K 2023-04-13 14:56:31,696 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=20.13 KB/20610 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 1642ms, sequenceid=1020, compaction requested=true 2023-04-13 14:56:31,696 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:31,696 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:31,696 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:31,696 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:56:31,696 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:31,696 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:56:31,697 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79126 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:56:31,697 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:56:31,697 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/A is initiating minor compaction (all files) 2023-04-13 14:56:31,697 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/B is initiating minor compaction (all files) 2023-04-13 14:56:31,697 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/A in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:31,697 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/B in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:31,698 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/177a2fa19bf340388d7b009ad556e017, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/e395780fb82347958cf3716ba4d79d50, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/110a8e2d146b49a3b2b55373d44fea58] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=77.3 K 2023-04-13 14:56:31,698 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/d48ab40c4b5a498ab66cb1c620ca9b04, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8edfc47949c4431a8fec96787fa4fc6e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/68f3bb84c7ce4ea485ea325013a91e1e] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=36.7 K 2023-04-13 14:56:31,698 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 177a2fa19bf340388d7b009ad556e017, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=969, earliestPutTs=1681397787795 2023-04-13 14:56:31,698 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting d48ab40c4b5a498ab66cb1c620ca9b04, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=969, earliestPutTs=1681397787795 2023-04-13 14:56:31,698 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting e395780fb82347958cf3716ba4d79d50, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1007, earliestPutTs=1681397787901 2023-04-13 14:56:31,698 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 8edfc47949c4431a8fec96787fa4fc6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1007, earliestPutTs=1681397787901 2023-04-13 14:56:31,698 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 110a8e2d146b49a3b2b55373d44fea58, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1020, earliestPutTs=1681397788496 2023-04-13 14:56:31,698 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] compactions.Compactor(207): Compacting 68f3bb84c7ce4ea485ea325013a91e1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1020, earliestPutTs=1681397788496 2023-04-13 14:56:31,704 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#B#compaction#610 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:31,708 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#A#compaction#611 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:31,720 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/09e8785ff3f540d7a2f70d8f045b30f8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/09e8785ff3f540d7a2f70d8f045b30f8 2023-04-13 14:56:31,724 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/A of 7f6fba518b666b8072f5bc2fa5e4a5d5 into 09e8785ff3f540d7a2f70d8f045b30f8(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:31,724 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:31,724 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/A, priority=13, startTime=1681397791696; duration=0sec 2023-04-13 14:56:31,724 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 14:56:31,724 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 14:56:31,725 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 14:56:31,725 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1912): 7f6fba518b666b8072f5bc2fa5e4a5d5/C is initiating minor compaction (all files) 2023-04-13 14:56:31,725 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f6fba518b666b8072f5bc2fa5e4a5d5/C in TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:31,725 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/40f82a8b2b484663a42b4737c1464248, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/87bfb321933649e38f7e12006d22ee04, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/7ffd83d408874ef68ff40dc90b709929] into tmpdir=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp, totalSize=36.7 K 2023-04-13 14:56:31,725 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 40f82a8b2b484663a42b4737c1464248, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=969, earliestPutTs=1681397787795 2023-04-13 14:56:31,725 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 87bfb321933649e38f7e12006d22ee04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1007, earliestPutTs=1681397787901 2023-04-13 14:56:31,726 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] compactions.Compactor(207): Compacting 7ffd83d408874ef68ff40dc90b709929, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1020, earliestPutTs=1681397788496 2023-04-13 14:56:31,730 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f6fba518b666b8072f5bc2fa5e4a5d5#C#compaction#612 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 14:56:32,113 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/ed216368543a4162990887e296c1c950 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/ed216368543a4162990887e296c1c950 2023-04-13 14:56:32,117 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/B of 7f6fba518b666b8072f5bc2fa5e4a5d5 into ed216368543a4162990887e296c1c950(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:32,117 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:32,117 INFO [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/B, priority=13, startTime=1681397791696; duration=0sec 2023-04-13 14:56:32,117 DEBUG [RS:0;jenkins-hbase4:43875-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:32,136 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/d288cc7c2def41419455b246bb41aa56 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/d288cc7c2def41419455b246bb41aa56 2023-04-13 14:56:32,140 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f6fba518b666b8072f5bc2fa5e4a5d5/C of 7f6fba518b666b8072f5bc2fa5e4a5d5 into d288cc7c2def41419455b246bb41aa56(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 14:56:32,140 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:32,140 INFO [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5., storeName=7f6fba518b666b8072f5bc2fa5e4a5d5/C, priority=13, startTime=1681397791696; duration=0sec 2023-04-13 14:56:32,140 DEBUG [RS:0;jenkins-hbase4:43875-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 14:56:38,595 DEBUG [Thread-3823] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e8654 to 127.0.0.1:62626 2023-04-13 14:56:38,596 DEBUG [Thread-3823] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:38,596 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 14:56:38,596 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 42 2023-04-13 14:56:38,596 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 470 2023-04-13 14:56:38,596 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 37 2023-04-13 14:56:38,596 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 326 2023-04-13 14:56:38,596 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(394): wrote 40 2023-04-13 14:56:38,596 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 14:56:38,596 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(398): read 4274 2023-04-13 14:56:38,596 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(398): read 4229 2023-04-13 14:56:38,596 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(398): read 4243 2023-04-13 14:56:38,596 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(398): read 4248 2023-04-13 14:56:38,596 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(398): read 4211 2023-04-13 14:56:38,596 INFO [Listener at localhost/37101] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 14:56:38,596 INFO [Listener at localhost/37101] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 14:56:38,596 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x470742aa to 127.0.0.1:62626 2023-04-13 14:56:38,596 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:38,597 INFO [Listener at localhost/37101] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 14:56:38,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 14:56:38,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:38,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 14:56:38,600 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397798600"}]},"ts":"1681397798600"} 2023-04-13 14:56:38,601 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 14:56:38,602 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 14:56:38,603 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f6fba518b666b8072f5bc2fa5e4a5d5, UNASSIGN}] 2023-04-13 14:56:38,604 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f6fba518b666b8072f5bc2fa5e4a5d5, UNASSIGN 2023-04-13 14:56:38,605 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=7f6fba518b666b8072f5bc2fa5e4a5d5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:38,605 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397798605"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681397798605"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681397798605"}]},"ts":"1681397798605"} 2023-04-13 14:56:38,605 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638}] 2023-04-13 14:56:38,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 14:56:38,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:38,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7f6fba518b666b8072f5bc2fa5e4a5d5, disabling compactions & flushes 2023-04-13 14:56:38,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:38,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:38,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. after waiting 0 ms 2023-04-13 14:56:38,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:38,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 7f6fba518b666b8072f5bc2fa5e4a5d5 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 14:56:38,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=A 2023-04-13 14:56:38,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:38,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=B 2023-04-13 14:56:38,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:38,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f6fba518b666b8072f5bc2fa5e4a5d5, store=C 2023-04-13 14:56:38,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 14:56:38,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 14:56:38,980 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-13 14:56:39,167 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304138151ac2d2ba84503881d8a827014ab54 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304138151ac2d2ba84503881d8a827014ab54 2023-04-13 14:56:39,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 14:56:39,572 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1030, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/987ac0585eaa4a73afff08238fc7caf5 2023-04-13 14:56:39,579 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1030 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/14a10d469872497494b2d646819c5085 2023-04-13 14:56:39,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 14:56:39,987 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1030 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/40a19e9bb7d04c5ab29b384d6a0e22df 2023-04-13 14:56:39,990 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/A/987ac0585eaa4a73afff08238fc7caf5 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/987ac0585eaa4a73afff08238fc7caf5 2023-04-13 14:56:39,993 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/987ac0585eaa4a73afff08238fc7caf5, entries=150, sequenceid=1030, filesize=25.4 K 2023-04-13 14:56:39,993 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/B/14a10d469872497494b2d646819c5085 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/14a10d469872497494b2d646819c5085 2023-04-13 14:56:39,996 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/14a10d469872497494b2d646819c5085, entries=150, sequenceid=1030, filesize=11.9 K 2023-04-13 14:56:39,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/.tmp/C/40a19e9bb7d04c5ab29b384d6a0e22df as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/40a19e9bb7d04c5ab29b384d6a0e22df 2023-04-13 14:56:39,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/40a19e9bb7d04c5ab29b384d6a0e22df, entries=150, sequenceid=1030, filesize=11.9 K 2023-04-13 14:56:40,000 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 7f6fba518b666b8072f5bc2fa5e4a5d5 in 1243ms, sequenceid=1030, compaction requested=false 2023-04-13 14:56:40,022 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/6c5ea36e7d5e483f90aaf42fa17eea93, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/4fa4795a9e3e4d5a9436db1b050dde28, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ef23692be430484a8a3f9bfd51a9c33b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2bea3ad4c2e3488797ca75b983c8414b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/3eaa196ef2524bf9be3d18858c33a8a6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/dd1839b084354bf3a3984bc534d1feaa, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/3e166f2d889f454b81e72c0af27fd2b4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/d3cd624fa126414bb61f5ce3cba6779b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70b80995c6414e138af645b0b61d3e3c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5b5b62429cb84914bbb15f9c929c8fe5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/207be9deabcc41ca8afb1ac5066be5e2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ce4e3a9b6bb14438b241c203712fb40c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/7f86114abf57477ca67b780a3dda4880, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/9118a2ea6d1f439fac0ffe3dc2a98577, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ecffca786aa048e4a98b60f1b7b54fa6, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/04a2fb821515475d970916a0910ccf6c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/e60df33ca8c64d508d3bc8a276ab9536, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/1f2914dc00bc4ce19d0db86977ddf8a3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/a4dd7133d4e6424488ce9d74b1fa7d6f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5ad6d17e50374a9ba7899c6e9ae59ef4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/d46905e9f4a44027af189b49cdf69529, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70551d01b0b54fc49bcfbbebe63ca5b4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70a0a2f7d1ee491898ae6cb1039d96a1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/49762fa6c2a24dec8ff127cb11382a8f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/4b20323d6f5744bcb66a4977e833b98b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/8db0ea6a4e4e45e6806e47bc600a434a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2b05fc04189248d3b8344f0c8c214c8f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2e7497d872d44f6192c704ab37165071, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5bbc4dc88ba142858696b346d2bf5f4f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/177a2fa19bf340388d7b009ad556e017, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/e395780fb82347958cf3716ba4d79d50, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/110a8e2d146b49a3b2b55373d44fea58] to archive 2023-04-13 14:56:40,023 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:56:40,025 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/6c5ea36e7d5e483f90aaf42fa17eea93 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/6c5ea36e7d5e483f90aaf42fa17eea93 2023-04-13 14:56:40,026 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/4fa4795a9e3e4d5a9436db1b050dde28 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/4fa4795a9e3e4d5a9436db1b050dde28 2023-04-13 14:56:40,027 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ef23692be430484a8a3f9bfd51a9c33b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ef23692be430484a8a3f9bfd51a9c33b 2023-04-13 14:56:40,028 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2bea3ad4c2e3488797ca75b983c8414b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2bea3ad4c2e3488797ca75b983c8414b 2023-04-13 14:56:40,029 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/3eaa196ef2524bf9be3d18858c33a8a6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/3eaa196ef2524bf9be3d18858c33a8a6 2023-04-13 14:56:40,030 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/dd1839b084354bf3a3984bc534d1feaa to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/dd1839b084354bf3a3984bc534d1feaa 2023-04-13 14:56:40,030 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/3e166f2d889f454b81e72c0af27fd2b4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/3e166f2d889f454b81e72c0af27fd2b4 2023-04-13 14:56:40,031 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/d3cd624fa126414bb61f5ce3cba6779b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/d3cd624fa126414bb61f5ce3cba6779b 2023-04-13 14:56:40,032 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70b80995c6414e138af645b0b61d3e3c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70b80995c6414e138af645b0b61d3e3c 2023-04-13 14:56:40,033 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5b5b62429cb84914bbb15f9c929c8fe5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5b5b62429cb84914bbb15f9c929c8fe5 2023-04-13 14:56:40,034 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/207be9deabcc41ca8afb1ac5066be5e2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/207be9deabcc41ca8afb1ac5066be5e2 2023-04-13 14:56:40,034 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ce4e3a9b6bb14438b241c203712fb40c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ce4e3a9b6bb14438b241c203712fb40c 2023-04-13 14:56:40,035 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/7f86114abf57477ca67b780a3dda4880 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/7f86114abf57477ca67b780a3dda4880 2023-04-13 14:56:40,036 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/9118a2ea6d1f439fac0ffe3dc2a98577 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/9118a2ea6d1f439fac0ffe3dc2a98577 2023-04-13 14:56:40,037 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ecffca786aa048e4a98b60f1b7b54fa6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/ecffca786aa048e4a98b60f1b7b54fa6 2023-04-13 14:56:40,038 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/04a2fb821515475d970916a0910ccf6c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/04a2fb821515475d970916a0910ccf6c 2023-04-13 14:56:40,038 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/e60df33ca8c64d508d3bc8a276ab9536 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/e60df33ca8c64d508d3bc8a276ab9536 2023-04-13 14:56:40,039 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/1f2914dc00bc4ce19d0db86977ddf8a3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/1f2914dc00bc4ce19d0db86977ddf8a3 2023-04-13 14:56:40,040 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/a4dd7133d4e6424488ce9d74b1fa7d6f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/a4dd7133d4e6424488ce9d74b1fa7d6f 2023-04-13 14:56:40,041 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5ad6d17e50374a9ba7899c6e9ae59ef4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5ad6d17e50374a9ba7899c6e9ae59ef4 2023-04-13 14:56:40,042 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/d46905e9f4a44027af189b49cdf69529 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/d46905e9f4a44027af189b49cdf69529 2023-04-13 14:56:40,042 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70551d01b0b54fc49bcfbbebe63ca5b4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70551d01b0b54fc49bcfbbebe63ca5b4 2023-04-13 14:56:40,043 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70a0a2f7d1ee491898ae6cb1039d96a1 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/70a0a2f7d1ee491898ae6cb1039d96a1 2023-04-13 14:56:40,044 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/49762fa6c2a24dec8ff127cb11382a8f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/49762fa6c2a24dec8ff127cb11382a8f 2023-04-13 14:56:40,045 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/4b20323d6f5744bcb66a4977e833b98b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/4b20323d6f5744bcb66a4977e833b98b 2023-04-13 14:56:40,046 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/8db0ea6a4e4e45e6806e47bc600a434a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/8db0ea6a4e4e45e6806e47bc600a434a 2023-04-13 14:56:40,047 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2b05fc04189248d3b8344f0c8c214c8f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2b05fc04189248d3b8344f0c8c214c8f 2023-04-13 14:56:40,047 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2e7497d872d44f6192c704ab37165071 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/2e7497d872d44f6192c704ab37165071 2023-04-13 14:56:40,048 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5bbc4dc88ba142858696b346d2bf5f4f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/5bbc4dc88ba142858696b346d2bf5f4f 2023-04-13 14:56:40,049 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/177a2fa19bf340388d7b009ad556e017 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/177a2fa19bf340388d7b009ad556e017 2023-04-13 14:56:40,050 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/e395780fb82347958cf3716ba4d79d50 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/e395780fb82347958cf3716ba4d79d50 2023-04-13 14:56:40,050 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/110a8e2d146b49a3b2b55373d44fea58 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/110a8e2d146b49a3b2b55373d44fea58 2023-04-13 14:56:40,053 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/a614ffd5241d4a54a5bc01da31cff56f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/4055fca866454e39adc5d6de31a43884, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/d7316b8d15924fe3906b684c1690cf3a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/89004ef2a1684bca8be476237134f1f3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/618fe22ba3df49089c13064144185bf1, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/e5626b02956c4d79bd065c28768651d2, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/da64db90e06147edadd5dd1872f7662d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/653ff3a529f14970949c9cb2fc713c89, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c1446c8073a44d10a7bd56d98e984abc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/1660bc04e00b4111b25cb2e2317cc7a5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/aff4e27b80f7423383a2c4478a8736ab, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/4a0c45f66df544099054ce033e05cb9d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c8ae2684a539459ea0c94ec7118a602c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/5bd0284e6f2648e3956fecfb15a052df, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/14631e34520a4d98aa3e84b4215b31c9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/bbd559cd43264193a97461cd308334a7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/369962ac82fb46b386fcb79e2cb64f59, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8665ac2ba4ad4ee394f0f7dcd545a9e8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/e7598e9baaf74b72a22dbc483a0fa14f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/6f56139f88d141d0b859bd2e89dded2a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/f38144990cf34ff897aec9f37d9c41ae, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/38ae8a1d34ba4a3cbb78ca4f612d2e5e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c0b54d356af0483ca7a405e1dc9fd5bc, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8690f4440b7a4fef90d2afe02c6d6ac7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/5a3dc4f6817844a5a19531f1cd40a613, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/1363a54274084d56895cad4e0323766c, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c5e0e787068d4ff2b8c2013041e16fa8, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/f0a3d724bd9d4560903ced888e4289a5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/d48ab40c4b5a498ab66cb1c620ca9b04, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/beb9680ab6614d32976e3c28879ef3ec, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8edfc47949c4431a8fec96787fa4fc6e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/68f3bb84c7ce4ea485ea325013a91e1e] to archive 2023-04-13 14:56:40,053 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:56:40,054 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/a614ffd5241d4a54a5bc01da31cff56f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/a614ffd5241d4a54a5bc01da31cff56f 2023-04-13 14:56:40,055 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/4055fca866454e39adc5d6de31a43884 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/4055fca866454e39adc5d6de31a43884 2023-04-13 14:56:40,056 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/d7316b8d15924fe3906b684c1690cf3a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/d7316b8d15924fe3906b684c1690cf3a 2023-04-13 14:56:40,057 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/89004ef2a1684bca8be476237134f1f3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/89004ef2a1684bca8be476237134f1f3 2023-04-13 14:56:40,058 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/618fe22ba3df49089c13064144185bf1 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/618fe22ba3df49089c13064144185bf1 2023-04-13 14:56:40,059 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/e5626b02956c4d79bd065c28768651d2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/e5626b02956c4d79bd065c28768651d2 2023-04-13 14:56:40,060 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/da64db90e06147edadd5dd1872f7662d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/da64db90e06147edadd5dd1872f7662d 2023-04-13 14:56:40,061 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/653ff3a529f14970949c9cb2fc713c89 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/653ff3a529f14970949c9cb2fc713c89 2023-04-13 14:56:40,061 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c1446c8073a44d10a7bd56d98e984abc to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c1446c8073a44d10a7bd56d98e984abc 2023-04-13 14:56:40,062 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/1660bc04e00b4111b25cb2e2317cc7a5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/1660bc04e00b4111b25cb2e2317cc7a5 2023-04-13 14:56:40,063 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/aff4e27b80f7423383a2c4478a8736ab to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/aff4e27b80f7423383a2c4478a8736ab 2023-04-13 14:56:40,064 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/4a0c45f66df544099054ce033e05cb9d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/4a0c45f66df544099054ce033e05cb9d 2023-04-13 14:56:40,065 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c8ae2684a539459ea0c94ec7118a602c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c8ae2684a539459ea0c94ec7118a602c 2023-04-13 14:56:40,066 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/5bd0284e6f2648e3956fecfb15a052df to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/5bd0284e6f2648e3956fecfb15a052df 2023-04-13 14:56:40,067 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/14631e34520a4d98aa3e84b4215b31c9 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/14631e34520a4d98aa3e84b4215b31c9 2023-04-13 14:56:40,067 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/bbd559cd43264193a97461cd308334a7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/bbd559cd43264193a97461cd308334a7 2023-04-13 14:56:40,068 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/369962ac82fb46b386fcb79e2cb64f59 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/369962ac82fb46b386fcb79e2cb64f59 2023-04-13 14:56:40,069 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8665ac2ba4ad4ee394f0f7dcd545a9e8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8665ac2ba4ad4ee394f0f7dcd545a9e8 2023-04-13 14:56:40,070 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/e7598e9baaf74b72a22dbc483a0fa14f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/e7598e9baaf74b72a22dbc483a0fa14f 2023-04-13 14:56:40,070 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/6f56139f88d141d0b859bd2e89dded2a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/6f56139f88d141d0b859bd2e89dded2a 2023-04-13 14:56:40,071 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/f38144990cf34ff897aec9f37d9c41ae to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/f38144990cf34ff897aec9f37d9c41ae 2023-04-13 14:56:40,072 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/38ae8a1d34ba4a3cbb78ca4f612d2e5e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/38ae8a1d34ba4a3cbb78ca4f612d2e5e 2023-04-13 14:56:40,073 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c0b54d356af0483ca7a405e1dc9fd5bc to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c0b54d356af0483ca7a405e1dc9fd5bc 2023-04-13 14:56:40,074 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8690f4440b7a4fef90d2afe02c6d6ac7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8690f4440b7a4fef90d2afe02c6d6ac7 2023-04-13 14:56:40,074 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/5a3dc4f6817844a5a19531f1cd40a613 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/5a3dc4f6817844a5a19531f1cd40a613 2023-04-13 14:56:40,075 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/1363a54274084d56895cad4e0323766c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/1363a54274084d56895cad4e0323766c 2023-04-13 14:56:40,076 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c5e0e787068d4ff2b8c2013041e16fa8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/c5e0e787068d4ff2b8c2013041e16fa8 2023-04-13 14:56:40,077 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/f0a3d724bd9d4560903ced888e4289a5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/f0a3d724bd9d4560903ced888e4289a5 2023-04-13 14:56:40,078 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/d48ab40c4b5a498ab66cb1c620ca9b04 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/d48ab40c4b5a498ab66cb1c620ca9b04 2023-04-13 14:56:40,078 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/beb9680ab6614d32976e3c28879ef3ec to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/beb9680ab6614d32976e3c28879ef3ec 2023-04-13 14:56:40,079 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8edfc47949c4431a8fec96787fa4fc6e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/8edfc47949c4431a8fec96787fa4fc6e 2023-04-13 14:56:40,080 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/68f3bb84c7ce4ea485ea325013a91e1e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/68f3bb84c7ce4ea485ea325013a91e1e 2023-04-13 14:56:40,082 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/58b9b3f7317342bc83b3d6d48ad17838, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/56294651dbe248ea8f12152dad922271, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/e56867305bc945ff9130f54bae9d8045, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/0ace186f878e4475a3030f598e800c2f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/22951d6e11ac4b38851e3cb5feedf135, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/373df13911d340feb89204ff4ebeb3d9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/804ee7008aef4bc1a272c0baefa6d56d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/ad7e59d52ab046ea9c5e7e2f4841e1a3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/8c96c4d0b421444dae0c3f2a454614f5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6e0ffb6e98504efab5f28f2cb5ef7248, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/913fd4d71e4f43a48481a5c23328dd3e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/90d612a26d5e455dba9b350014d5ad33, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/5bdacb8d7a3d44d6ad0133bcd21c448e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/552bfdf7b4d144d9b8f5b3f7a5aa4860, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/7c79fd38e28545a98b5a76088580f85a, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/9290b4f8ec004061933713b6ee03fbd3, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b818a76702f249279767dc3d5c42d615, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b6f493632f3b4ed0b134262ab99e8f4d, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/1d4faf94de244f4580ef98f85f78dae9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6cc46e00499c472a88fdd1c958d2916f, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/670e73343e3048a188f6ba00bbee1734, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/901c1bf76d63469092d32c4e60cfb7f7, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/ac7bd67454574b4594906a2163482ff4, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/be9a75a69963458ab29e5d47069af1d9, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b84ad2945fd34c2786cee7aff41c3e60, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6bcc629b80b340028b508bf4cbb612a5, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/f2c6c14bf9ff4fc3929c10f2b1a0404b, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/460d1faa61c64dddb8ad5e0faeb9e87e, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/40f82a8b2b484663a42b4737c1464248, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/5a6de4b94c7746cf9e8173afc4df7700, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/87bfb321933649e38f7e12006d22ee04, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/7ffd83d408874ef68ff40dc90b709929] to archive 2023-04-13 14:56:40,083 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 14:56:40,084 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/58b9b3f7317342bc83b3d6d48ad17838 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/58b9b3f7317342bc83b3d6d48ad17838 2023-04-13 14:56:40,085 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/56294651dbe248ea8f12152dad922271 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/56294651dbe248ea8f12152dad922271 2023-04-13 14:56:40,086 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/e56867305bc945ff9130f54bae9d8045 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/e56867305bc945ff9130f54bae9d8045 2023-04-13 14:56:40,087 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/0ace186f878e4475a3030f598e800c2f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/0ace186f878e4475a3030f598e800c2f 2023-04-13 14:56:40,088 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/22951d6e11ac4b38851e3cb5feedf135 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/22951d6e11ac4b38851e3cb5feedf135 2023-04-13 14:56:40,089 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/373df13911d340feb89204ff4ebeb3d9 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/373df13911d340feb89204ff4ebeb3d9 2023-04-13 14:56:40,090 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/804ee7008aef4bc1a272c0baefa6d56d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/804ee7008aef4bc1a272c0baefa6d56d 2023-04-13 14:56:40,091 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/ad7e59d52ab046ea9c5e7e2f4841e1a3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/ad7e59d52ab046ea9c5e7e2f4841e1a3 2023-04-13 14:56:40,091 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/8c96c4d0b421444dae0c3f2a454614f5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/8c96c4d0b421444dae0c3f2a454614f5 2023-04-13 14:56:40,092 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6e0ffb6e98504efab5f28f2cb5ef7248 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6e0ffb6e98504efab5f28f2cb5ef7248 2023-04-13 14:56:40,093 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/913fd4d71e4f43a48481a5c23328dd3e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/913fd4d71e4f43a48481a5c23328dd3e 2023-04-13 14:56:40,094 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/90d612a26d5e455dba9b350014d5ad33 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/90d612a26d5e455dba9b350014d5ad33 2023-04-13 14:56:40,094 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/5bdacb8d7a3d44d6ad0133bcd21c448e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/5bdacb8d7a3d44d6ad0133bcd21c448e 2023-04-13 14:56:40,095 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/552bfdf7b4d144d9b8f5b3f7a5aa4860 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/552bfdf7b4d144d9b8f5b3f7a5aa4860 2023-04-13 14:56:40,096 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/7c79fd38e28545a98b5a76088580f85a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/7c79fd38e28545a98b5a76088580f85a 2023-04-13 14:56:40,096 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/9290b4f8ec004061933713b6ee03fbd3 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/9290b4f8ec004061933713b6ee03fbd3 2023-04-13 14:56:40,097 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b818a76702f249279767dc3d5c42d615 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b818a76702f249279767dc3d5c42d615 2023-04-13 14:56:40,098 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b6f493632f3b4ed0b134262ab99e8f4d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b6f493632f3b4ed0b134262ab99e8f4d 2023-04-13 14:56:40,099 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/1d4faf94de244f4580ef98f85f78dae9 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/1d4faf94de244f4580ef98f85f78dae9 2023-04-13 14:56:40,100 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6cc46e00499c472a88fdd1c958d2916f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6cc46e00499c472a88fdd1c958d2916f 2023-04-13 14:56:40,100 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/670e73343e3048a188f6ba00bbee1734 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/670e73343e3048a188f6ba00bbee1734 2023-04-13 14:56:40,101 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/901c1bf76d63469092d32c4e60cfb7f7 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/901c1bf76d63469092d32c4e60cfb7f7 2023-04-13 14:56:40,102 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/ac7bd67454574b4594906a2163482ff4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/ac7bd67454574b4594906a2163482ff4 2023-04-13 14:56:40,103 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/be9a75a69963458ab29e5d47069af1d9 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/be9a75a69963458ab29e5d47069af1d9 2023-04-13 14:56:40,104 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b84ad2945fd34c2786cee7aff41c3e60 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/b84ad2945fd34c2786cee7aff41c3e60 2023-04-13 14:56:40,104 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6bcc629b80b340028b508bf4cbb612a5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/6bcc629b80b340028b508bf4cbb612a5 2023-04-13 14:56:40,105 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/f2c6c14bf9ff4fc3929c10f2b1a0404b to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/f2c6c14bf9ff4fc3929c10f2b1a0404b 2023-04-13 14:56:40,106 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/460d1faa61c64dddb8ad5e0faeb9e87e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/460d1faa61c64dddb8ad5e0faeb9e87e 2023-04-13 14:56:40,107 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/40f82a8b2b484663a42b4737c1464248 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/40f82a8b2b484663a42b4737c1464248 2023-04-13 14:56:40,108 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/5a6de4b94c7746cf9e8173afc4df7700 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/5a6de4b94c7746cf9e8173afc4df7700 2023-04-13 14:56:40,108 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/87bfb321933649e38f7e12006d22ee04 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/87bfb321933649e38f7e12006d22ee04 2023-04-13 14:56:40,109 DEBUG [StoreCloser-TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/7ffd83d408874ef68ff40dc90b709929 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/7ffd83d408874ef68ff40dc90b709929 2023-04-13 14:56:40,113 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/recovered.edits/1033.seqid, newMaxSeqId=1033, maxSeqId=4 2023-04-13 14:56:40,114 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5. 2023-04-13 14:56:40,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7f6fba518b666b8072f5bc2fa5e4a5d5: 2023-04-13 14:56:40,115 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:40,116 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=7f6fba518b666b8072f5bc2fa5e4a5d5, regionState=CLOSED 2023-04-13 14:56:40,116 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681397800116"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681397800116"}]},"ts":"1681397800116"} 2023-04-13 14:56:40,117 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-13 14:56:40,118 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 7f6fba518b666b8072f5bc2fa5e4a5d5, server=jenkins-hbase4.apache.org,43875,1681397617638 in 1.5110 sec 2023-04-13 14:56:40,118 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-13 14:56:40,118 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f6fba518b666b8072f5bc2fa5e4a5d5, UNASSIGN in 1.5150 sec 2023-04-13 14:56:40,119 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681397800119"}]},"ts":"1681397800119"} 2023-04-13 14:56:40,120 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 14:56:40,125 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 14:56:40,126 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5280 sec 2023-04-13 14:56:40,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 14:56:40,705 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-13 14:56:40,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 14:56:40,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:40,708 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:40,708 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:40,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-13 14:56:40,711 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:40,713 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C, FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/recovered.edits] 2023-04-13 14:56:40,715 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/09e8785ff3f540d7a2f70d8f045b30f8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/09e8785ff3f540d7a2f70d8f045b30f8 2023-04-13 14:56:40,716 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/987ac0585eaa4a73afff08238fc7caf5 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/A/987ac0585eaa4a73afff08238fc7caf5 2023-04-13 14:56:40,718 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/14a10d469872497494b2d646819c5085 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/14a10d469872497494b2d646819c5085 2023-04-13 14:56:40,719 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/ed216368543a4162990887e296c1c950 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/B/ed216368543a4162990887e296c1c950 2023-04-13 14:56:40,720 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/40a19e9bb7d04c5ab29b384d6a0e22df to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/40a19e9bb7d04c5ab29b384d6a0e22df 2023-04-13 14:56:40,721 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/d288cc7c2def41419455b246bb41aa56 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/C/d288cc7c2def41419455b246bb41aa56 2023-04-13 14:56:40,723 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/recovered.edits/1033.seqid to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5/recovered.edits/1033.seqid 2023-04-13 14:56:40,724 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/.tmp/data/default/TestAcidGuarantees/7f6fba518b666b8072f5bc2fa5e4a5d5 2023-04-13 14:56:40,724 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 14:56:40,724 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 14:56:40,725 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-13 14:56:40,727 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413013244a3179a4ca2a839e1bf3fe8ffeb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413013244a3179a4ca2a839e1bf3fe8ffeb 2023-04-13 14:56:40,728 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041305a547d479a9479e9f25ec2dc2ff07bf to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041305a547d479a9479e9f25ec2dc2ff07bf 2023-04-13 14:56:40,729 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304130756eba6cd1f42f29f570ec36b445ddb to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304130756eba6cd1f42f29f570ec36b445ddb 2023-04-13 14:56:40,730 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304131fbf6dd0f9014a4c95c64f503b3f2503 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304131fbf6dd0f9014a4c95c64f503b3f2503 2023-04-13 14:56:40,731 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041326c134b00ff7495fbc9295c17d7cd7bf to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041326c134b00ff7495fbc9295c17d7cd7bf 2023-04-13 14:56:40,732 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413279e2106c3d141e18019cfd483f3cd70 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413279e2106c3d141e18019cfd483f3cd70 2023-04-13 14:56:40,732 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041330b4ba1e8bb640f7a3f7493aaef6511e to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041330b4ba1e8bb640f7a3f7493aaef6511e 2023-04-13 14:56:40,733 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133891192dc0bb4f8d8c8792599372c7cf to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133891192dc0bb4f8d8c8792599372c7cf 2023-04-13 14:56:40,734 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133b0cd353c8b14ec28cdeedf81afd2a87 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133b0cd353c8b14ec28cdeedf81afd2a87 2023-04-13 14:56:40,735 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134402d421809e473d9f0c60585f8f34d8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134402d421809e473d9f0c60585f8f34d8 2023-04-13 14:56:40,736 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304135aed7df4d01b46b5aa0f88a21ad19fb2 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304135aed7df4d01b46b5aa0f88a21ad19fb2 2023-04-13 14:56:40,737 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304135d572185edf74f04be98f72ea498c1ab to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304135d572185edf74f04be98f72ea498c1ab 2023-04-13 14:56:40,737 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136793eb4cca03448ba55b6140ee132194 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136793eb4cca03448ba55b6140ee132194 2023-04-13 14:56:40,738 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304138151ac2d2ba84503881d8a827014ab54 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304138151ac2d2ba84503881d8a827014ab54 2023-04-13 14:56:40,739 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304138e05875ec4a94c1b99d689d344c9329d to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304138e05875ec4a94c1b99d689d344c9329d 2023-04-13 14:56:40,740 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b986939c336b487b89cc1165995a29a8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b986939c336b487b89cc1165995a29a8 2023-04-13 14:56:40,741 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ba4dd3f6ac9041b9b0f3e307eb2b0ca0 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ba4dd3f6ac9041b9b0f3e307eb2b0ca0 2023-04-13 14:56:40,742 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c4cee690218a439f9dac77a694ff8474 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c4cee690218a439f9dac77a694ff8474 2023-04-13 14:56:40,742 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c80acfebfc304109a00e31145fe9f79c to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c80acfebfc304109a00e31145fe9f79c 2023-04-13 14:56:40,743 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413cbb0b9fb8cf149eead353424b1b22ef4 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413cbb0b9fb8cf149eead353424b1b22ef4 2023-04-13 14:56:40,744 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413cdae2be7c42743f3ad20289489c47527 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413cdae2be7c42743f3ad20289489c47527 2023-04-13 14:56:40,745 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413cdef4e3b6db64635b07e07c019f45ce6 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413cdef4e3b6db64635b07e07c019f45ce6 2023-04-13 14:56:40,746 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d56dbd410c8748f4b0bfa94cb075b92a to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d56dbd410c8748f4b0bfa94cb075b92a 2023-04-13 14:56:40,747 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e1230515141445108c0f1e523201c1e8 to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e1230515141445108c0f1e523201c1e8 2023-04-13 14:56:40,748 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e89caafdc4b24ba3af0601638ef6391f to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e89caafdc4b24ba3af0601638ef6391f 2023-04-13 14:56:40,748 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ecd84892f00d4fe3a948d9587a9958ed to hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ecd84892f00d4fe3a948d9587a9958ed 2023-04-13 14:56:40,749 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 14:56:40,750 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:40,752 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 14:56:40,753 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 14:56:40,754 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:40,754 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 14:56:40,754 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681397800754"}]},"ts":"9223372036854775807"} 2023-04-13 14:56:40,755 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 14:56:40,755 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7f6fba518b666b8072f5bc2fa5e4a5d5, NAME => 'TestAcidGuarantees,,1681397765604.7f6fba518b666b8072f5bc2fa5e4a5d5.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 14:56:40,755 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 14:56:40,755 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681397800755"}]},"ts":"9223372036854775807"} 2023-04-13 14:56:40,756 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 14:56:40,758 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 14:56:40,759 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 53 msec 2023-04-13 14:56:40,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46829] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-13 14:56:40,810 INFO [Listener at localhost/37101] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-13 14:56:40,821 INFO [Listener at localhost/37101] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 229), OpenFileDescriptor=535 (was 592), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=403 (was 448), ProcessCount=169 (was 169), AvailableMemoryMB=1229 (was 1141) - AvailableMemoryMB LEAK? - 2023-04-13 14:56:40,821 INFO [Listener at localhost/37101] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-13 14:56:40,821 INFO [Listener at localhost/37101] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 14:56:40,822 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69cada93 to 127.0.0.1:62626 2023-04-13 14:56:40,822 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:40,822 DEBUG [Listener at localhost/37101] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-13 14:56:40,822 DEBUG [Listener at localhost/37101] util.JVMClusterUtil(257): Found active master hash=821109370, stopped=false 2023-04-13 14:56:40,822 INFO [Listener at localhost/37101] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,46829,1681397616434 2023-04-13 14:56:40,824 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-13 14:56:40,825 INFO [Listener at localhost/37101] procedure2.ProcedureExecutor(629): Stopping 2023-04-13 14:56:40,825 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 14:56:40,825 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-13 14:56:40,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-13 14:56:40,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-13 14:56:40,825 DEBUG [Listener at localhost/37101] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0307b598 to 127.0.0.1:62626 2023-04-13 14:56:40,825 DEBUG [Listener at localhost/37101] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:40,825 INFO [Listener at localhost/37101] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43875,1681397617638' ***** 2023-04-13 14:56:40,826 INFO [Listener at localhost/37101] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-13 14:56:40,826 INFO [RS:0;jenkins-hbase4:43875] regionserver.HeapMemoryManager(220): Stopping 2023-04-13 14:56:40,826 INFO [RS:0;jenkins-hbase4:43875] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-13 14:56:40,826 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-13 14:56:40,826 INFO [RS:0;jenkins-hbase4:43875] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-13 14:56:40,826 INFO [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(3304): Received CLOSE for f80e0880801c37895d7e91109ae26c79 2023-04-13 14:56:40,827 INFO [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:40,827 DEBUG [RS:0;jenkins-hbase4:43875] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3eff0014 to 127.0.0.1:62626 2023-04-13 14:56:40,827 DEBUG [RS:0;jenkins-hbase4:43875] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:40,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f80e0880801c37895d7e91109ae26c79, disabling compactions & flushes 2023-04-13 14:56:40,828 INFO [RS:0;jenkins-hbase4:43875] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-13 14:56:40,828 INFO [RS:0;jenkins-hbase4:43875] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-13 14:56:40,828 INFO [RS:0;jenkins-hbase4:43875] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-13 14:56:40,828 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79. 2023-04-13 14:56:40,828 INFO [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-13 14:56:40,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79. 2023-04-13 14:56:40,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79. after waiting 0 ms 2023-04-13 14:56:40,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79. 2023-04-13 14:56:40,828 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing f80e0880801c37895d7e91109ae26c79 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-13 14:56:40,828 INFO [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-13 14:56:40,828 DEBUG [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, f80e0880801c37895d7e91109ae26c79=hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79.} 2023-04-13 14:56:40,829 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-13 14:56:40,829 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-13 14:56:40,829 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-13 14:56:40,829 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-13 14:56:40,829 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-13 14:56:40,829 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-13 14:56:40,830 DEBUG [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1505): Waiting on 1588230740, f80e0880801c37895d7e91109ae26c79 2023-04-13 14:56:40,845 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-13 14:56:40,944 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-13 14:56:40,945 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-13 14:56:41,030 DEBUG [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1505): Waiting on 1588230740, f80e0880801c37895d7e91109ae26c79 2023-04-13 14:56:41,231 DEBUG [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1505): Waiting on 1588230740, f80e0880801c37895d7e91109ae26c79 2023-04-13 14:56:41,241 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/.tmp/info/ef256d6a4c974690b22f16580bea2445 2023-04-13 14:56:41,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/namespace/f80e0880801c37895d7e91109ae26c79/.tmp/info/40bbe37284f94725b8ee66109511cfb8 2023-04-13 14:56:41,245 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/namespace/f80e0880801c37895d7e91109ae26c79/.tmp/info/40bbe37284f94725b8ee66109511cfb8 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/namespace/f80e0880801c37895d7e91109ae26c79/info/40bbe37284f94725b8ee66109511cfb8 2023-04-13 14:56:41,245 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ef256d6a4c974690b22f16580bea2445 2023-04-13 14:56:41,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/namespace/f80e0880801c37895d7e91109ae26c79/info/40bbe37284f94725b8ee66109511cfb8, entries=2, sequenceid=6, filesize=4.8 K 2023-04-13 14:56:41,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for f80e0880801c37895d7e91109ae26c79 in 421ms, sequenceid=6, compaction requested=false 2023-04-13 14:56:41,253 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/.tmp/rep_barrier/8ec7996ae59847629066ded27069342d 2023-04-13 14:56:41,255 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/namespace/f80e0880801c37895d7e91109ae26c79/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-13 14:56:41,256 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79. 2023-04-13 14:56:41,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f80e0880801c37895d7e91109ae26c79: 2023-04-13 14:56:41,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681397620336.f80e0880801c37895d7e91109ae26c79. 2023-04-13 14:56:41,256 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8ec7996ae59847629066ded27069342d 2023-04-13 14:56:41,431 DEBUG [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-13 14:56:41,631 DEBUG [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-13 14:56:41,661 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/.tmp/table/7747675c6d07462b8c98085f7748cc75 2023-04-13 14:56:41,664 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7747675c6d07462b8c98085f7748cc75 2023-04-13 14:56:41,665 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/.tmp/info/ef256d6a4c974690b22f16580bea2445 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/info/ef256d6a4c974690b22f16580bea2445 2023-04-13 14:56:41,667 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ef256d6a4c974690b22f16580bea2445 2023-04-13 14:56:41,667 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/info/ef256d6a4c974690b22f16580bea2445, entries=22, sequenceid=93, filesize=7.1 K 2023-04-13 14:56:41,669 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/.tmp/rep_barrier/8ec7996ae59847629066ded27069342d as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/rep_barrier/8ec7996ae59847629066ded27069342d 2023-04-13 14:56:41,671 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8ec7996ae59847629066ded27069342d 2023-04-13 14:56:41,671 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/rep_barrier/8ec7996ae59847629066ded27069342d, entries=6, sequenceid=93, filesize=5.5 K 2023-04-13 14:56:41,672 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/.tmp/table/7747675c6d07462b8c98085f7748cc75 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/table/7747675c6d07462b8c98085f7748cc75 2023-04-13 14:56:41,674 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7747675c6d07462b8c98085f7748cc75 2023-04-13 14:56:41,674 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/table/7747675c6d07462b8c98085f7748cc75, entries=9, sequenceid=93, filesize=5.5 K 2023-04-13 14:56:41,675 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 846ms, sequenceid=93, compaction requested=false 2023-04-13 14:56:41,682 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-13 14:56:41,682 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-13 14:56:41,683 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-13 14:56:41,683 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-13 14:56:41,683 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-13 14:56:41,832 INFO [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43875,1681397617638; all regions closed. 2023-04-13 14:56:41,839 DEBUG [RS:0;jenkins-hbase4:43875] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/oldWALs 2023-04-13 14:56:41,839 INFO [RS:0;jenkins-hbase4:43875] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43875%2C1681397617638.meta:.meta(num 1681397620062) 2023-04-13 14:56:41,843 DEBUG [RS:0;jenkins-hbase4:43875] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/oldWALs 2023-04-13 14:56:41,843 INFO [RS:0;jenkins-hbase4:43875] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43875%2C1681397617638:(num 1681397619111) 2023-04-13 14:56:41,843 DEBUG [RS:0;jenkins-hbase4:43875] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:41,843 INFO [RS:0;jenkins-hbase4:43875] regionserver.LeaseManager(133): Closed leases 2023-04-13 14:56:41,844 INFO [RS:0;jenkins-hbase4:43875] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-13 14:56:41,844 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-13 14:56:41,845 INFO [RS:0;jenkins-hbase4:43875] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43875 2023-04-13 14:56:41,850 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43875,1681397617638 2023-04-13 14:56:41,850 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-13 14:56:41,850 ERROR [Listener at localhost/37101-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@43dc4ac2 rejected from java.util.concurrent.ThreadPoolExecutor@47e3c661[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2386] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 14:56:41,851 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-13 14:56:41,851 ERROR [Listener at localhost/37101-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@12821691 rejected from java.util.concurrent.ThreadPoolExecutor@47e3c661[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2386] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 14:56:41,852 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,43875,1681397617638] 2023-04-13 14:56:41,853 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,43875,1681397617638; numProcessing=1 2023-04-13 14:56:41,854 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,43875,1681397617638 already deleted, retry=false 2023-04-13 14:56:41,854 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,43875,1681397617638 expired; onlineServers=0 2023-04-13 14:56:41,854 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46829,1681397616434' ***** 2023-04-13 14:56:41,854 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-13 14:56:41,854 DEBUG [M:0;jenkins-hbase4:46829] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71cc2841, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-13 14:56:41,854 INFO [M:0;jenkins-hbase4:46829] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46829,1681397616434 2023-04-13 14:56:41,854 INFO [M:0;jenkins-hbase4:46829] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46829,1681397616434; all regions closed. 2023-04-13 14:56:41,854 DEBUG [M:0;jenkins-hbase4:46829] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 14:56:41,854 DEBUG [M:0;jenkins-hbase4:46829] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-13 14:56:41,854 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-13 14:56:41,855 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681397618783] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681397618783,5,FailOnTimeoutGroup] 2023-04-13 14:56:41,855 DEBUG [M:0;jenkins-hbase4:46829] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-13 14:56:41,855 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681397618779] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681397618779,5,FailOnTimeoutGroup] 2023-04-13 14:56:41,855 INFO [M:0;jenkins-hbase4:46829] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-13 14:56:41,855 INFO [M:0;jenkins-hbase4:46829] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-13 14:56:41,855 INFO [M:0;jenkins-hbase4:46829] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-13 14:56:41,855 DEBUG [M:0;jenkins-hbase4:46829] master.HMaster(1512): Stopping service threads 2023-04-13 14:56:41,855 INFO [M:0;jenkins-hbase4:46829] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-13 14:56:41,856 ERROR [M:0;jenkins-hbase4:46829] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-13 14:56:41,856 INFO [M:0;jenkins-hbase4:46829] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-13 14:56:41,856 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-13 14:56:41,856 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-13 14:56:41,856 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 14:56:41,857 DEBUG [M:0;jenkins-hbase4:46829] zookeeper.ZKUtil(398): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-13 14:56:41,857 WARN [M:0;jenkins-hbase4:46829] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-13 14:56:41,857 INFO [M:0;jenkins-hbase4:46829] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-13 14:56:41,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-13 14:56:41,857 INFO [M:0;jenkins-hbase4:46829] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-13 14:56:41,857 DEBUG [M:0;jenkins-hbase4:46829] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-13 14:56:41,857 INFO [M:0;jenkins-hbase4:46829] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 14:56:41,857 DEBUG [M:0;jenkins-hbase4:46829] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 14:56:41,857 DEBUG [M:0;jenkins-hbase4:46829] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-13 14:56:41,857 DEBUG [M:0;jenkins-hbase4:46829] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 14:56:41,857 INFO [M:0;jenkins-hbase4:46829] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.07 KB heapSize=393.84 KB 2023-04-13 14:56:41,953 INFO [RS:0;jenkins-hbase4:43875] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43875,1681397617638; zookeeper connection closed. 2023-04-13 14:56:41,953 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-13 14:56:41,954 ERROR [Listener at localhost/37101-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6a20cc83 rejected from java.util.concurrent.ThreadPoolExecutor@47e3c661[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2386] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 14:56:41,954 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@20675b6] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@20675b6 2023-04-13 14:56:41,954 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): regionserver:43875-0x1017e0375af0001, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-13 14:56:41,955 ERROR [Listener at localhost/37101-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@60fa0be0 rejected from java.util.concurrent.ThreadPoolExecutor@47e3c661[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2386] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 14:56:41,955 INFO [Listener at localhost/37101] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-13 14:56:42,266 INFO [M:0;jenkins-hbase4:46829] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.07 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/a971b6c4ebcc4ffbbf3cd252fe104356 2023-04-13 14:56:42,270 INFO [M:0;jenkins-hbase4:46829] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a971b6c4ebcc4ffbbf3cd252fe104356 2023-04-13 14:56:42,271 DEBUG [M:0;jenkins-hbase4:46829] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/a971b6c4ebcc4ffbbf3cd252fe104356 as hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/a971b6c4ebcc4ffbbf3cd252fe104356 2023-04-13 14:56:42,273 INFO [M:0;jenkins-hbase4:46829] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a971b6c4ebcc4ffbbf3cd252fe104356 2023-04-13 14:56:42,274 INFO [M:0;jenkins-hbase4:46829] regionserver.HStore(1080): Added hdfs://localhost:38971/user/jenkins/test-data/b98ab9e8-3d6e-60f7-9d06-d7761a944e3b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/a971b6c4ebcc4ffbbf3cd252fe104356, entries=65, sequenceid=579, filesize=24.7 K 2023-04-13 14:56:42,274 INFO [M:0;jenkins-hbase4:46829] regionserver.HRegion(2948): Finished flush of dataSize ~345.07 KB/353356, heapSize ~393.82 KB/403272, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 417ms, sequenceid=579, compaction requested=false 2023-04-13 14:56:42,276 INFO [M:0;jenkins-hbase4:46829] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 14:56:42,276 DEBUG [M:0;jenkins-hbase4:46829] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-13 14:56:42,279 INFO [M:0;jenkins-hbase4:46829] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-13 14:56:42,279 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-13 14:56:42,280 INFO [M:0;jenkins-hbase4:46829] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46829 2023-04-13 14:56:42,282 DEBUG [M:0;jenkins-hbase4:46829] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,46829,1681397616434 already deleted, retry=false 2023-04-13 14:56:42,384 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-13 14:56:42,384 INFO [M:0;jenkins-hbase4:46829] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46829,1681397616434; zookeeper connection closed. 2023-04-13 14:56:42,384 ERROR [Listener at localhost/37101-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6c100e60 rejected from java.util.concurrent.ThreadPoolExecutor@59558d26[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3982] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 14:56:42,384 DEBUG [Listener at localhost/37101-EventThread] zookeeper.ZKWatcher(600): master:46829-0x1017e0375af0000, quorum=127.0.0.1:62626, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-13 14:56:42,384 ERROR [Listener at localhost/37101-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5b9774f3 rejected from java.util.concurrent.ThreadPoolExecutor@59558d26[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3982] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 14:56:42,386 WARN [Listener at localhost/37101] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-13 14:56:42,389 INFO [Listener at localhost/37101] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-13 14:56:42,497 WARN [BP-23623483-172.31.14.131-1681397613026 heartbeating to localhost/127.0.0.1:38971] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-13 14:56:42,497 WARN [BP-23623483-172.31.14.131-1681397613026 heartbeating to localhost/127.0.0.1:38971] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-23623483-172.31.14.131-1681397613026 (Datanode Uuid 3aa5f5e1-9f33-4bd6-908d-6f9dfe621103) service to localhost/127.0.0.1:38971 2023-04-13 14:56:42,500 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/cluster_688f3769-b6e4-94a5-b788-510dc932696f/dfs/data/data1/current/BP-23623483-172.31.14.131-1681397613026] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-13 14:56:42,500 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/00d9e268-2234-0837-0581-67e3c5e75d82/cluster_688f3769-b6e4-94a5-b788-510dc932696f/dfs/data/data2/current/BP-23623483-172.31.14.131-1681397613026] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-13 14:56:42,538 INFO [Listener at localhost/37101] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-13 14:56:42,706 INFO [Listener at localhost/37101] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-13 14:56:42,875 INFO [Listener at localhost/37101] hbase.HBaseTestingUtility(1293): Minicluster is down